Einstein Toolkit Documentation - User contributions [en]

Thorns we know of

2012-04-24T03:52:58Z

Barry.wardell:

There are more public thorns around than just those in the Einstein Toolkit. This might be because the thorn does not satisfy our quality standards or because it is not yet clear if the thorn would be used by a number of users.

If you have thorn that you would make public and maybe entually see included in the Einstein Toolkit, you are encouraged to list it here, together with a short description of what it does. Our hope is to provide a list of (most of) the available Cactus thorns. Having a thorn listed here also makes it easier for us to estimate how much interest there is in it.

Source code repositories are vastly preferred over just source tarballs since they make it much easier for your users to contribute back changes.
If you are not yourself able to host a repository you might consider one of the public source code hosters such as those listed on [https://en.wikipedia.org/wiki/Comparison_of_open_source_software_hosting_facilities Wikipedia]. Some of [mailto:users@einsteintoolkit.org us] have used GitHub in the past as well as BitBucket (but see ticket [https://trac.einsteintoolkit.org/ticket/697 #697] for issues with it).

{| class="wikitable" border="1"
|-
| Thorn name
| Description
| Author(s)
| Licence
| Download instructions
|-
| Outflow
| Computes the flow of rest mass through a SphericalSurface, allows extra variables to be interpolated onto the surface
| Roland Haas, Tanja Bode
| GPL v2+
| svn co https://svn.einsteintoolkit.org/incoming/Outflow
|-
| SphericalSlice
| Provides infrastructure to interpolate grid functions onto sphere, to compute intergrals as well as IO facilities.
| Christian Reisswig
| unknown
| svn co https://svn.einsteintoolkit.org/incoming/SphericalSlice
|-
| ADMDerivatives
| Computes time and space derivatives of the ADMBase variables
| Christian Reisswig
| GNU
| svn co https://svn.einsteintoolkit.org/incoming/ADMDerivatives
|-
| SphericalHarmonicReconASCII
| Computes boundary data for CCE using SphericalSlice and ADMDerivatives
| Christian Reisswig
| unknown
| svn co https://svn.einsteintoolkit.org/incoming/PITTNullCode/SphericalHarmonicReconASCII/
|-
| EinsteinExact
| Computes exact initial data for the ADMBase variables
| Barry Wardell and Ian Hinder
| GPL
| git clone --recursive git://github.com/barrywardell/EinsteinExact
|-
|}

GitSuperRepoAdminGuide

2011-07-12T10:26:56Z

Barry.wardell: /* Setting up Gitolite */

[[Category:GitSuperRepo]]

= Local mirrors =
The modules in a Cactus tree typically constitute a large number of repositories in various version control systems from wide number of sources. While GetComponents makes it get everything from their original locations, it may be still desirable to create local mirrors of these modules for several reasons:
* '''Speed''' - Some sources may be on the other side of the world so having a nearby local mirror can dramatically reduce the time to access them.
* '''Privacy''' - You may want to develop a new feature privately within your group before publicly releasing it. If this feature is based on an existing module then having a local mirror means you can create local private development branch which can later be easily merged and pushed upstream.
* '''Convenience''' - With remote repositories in a variety of version control systems, it is easy to get confused. Having local mirrors of these in a single version control system (i.e. git) simplifies things so that only a single tool is needed. In the case where the remote repository is in SVN local git mirrors also provides the advantage of offering the additional features git provides over svn.

This section describes how to setup, host and administer local mirrors of remote repositories.

== Creating repository mirrors ==

== Adding repository mirrors ==

== Removing repository mirrors ==

= Hosting mirrors on a server =

Rather than everyone creating their own local mirrors which they have to worry about keeping up-to-date, it makes sense to have a central server do so. In this section, we describe how to setup a server using [https://github.com/sitaramc/gitolite Gitolite].

== Setting up Gitolite ==
Follow the Gitolite [http://sitaramc.github.com/gitolite/doc/1-INSTALL.html#_root_method installation instructions]:

On the server (as root):
useradd -d /var/git -s /bin/bash -m git
git clone git://github.com/sitaramc/gitolite
cd gitolite
src/gl-system-install
su - git
gl-setup /tmp/admin.pub

On your workstation:
git clone git@server:gitolite-admin

== Permissions ==

== Optional Extras ==
=== Blocking pushes of unnecessary merge commits ===

While working on a change in a git repository, it often happens that while you make local commits someone else pushes their own changes to the central repository. If you then try to push your changes, your push will be rejected as a non-fast-forward commit. The easiest solution is to pull the new changes from the server before pushing. However, this will introduce a 'merge commit' in the process. These merge commits rapidly build up and clutter up the history of the repository. For this reason, instead of pulling and introducing a merge commit you should do one of two things:
* Rebase your changes on top of the central repository. This can be achieved using 'git pull --rebase'.
* If your changes are significant, you can create a new named branch for them and merge that branch into the branch from the central repository.
In an ideal world, everyone would always follow these rules. However, in practice it is easy to forget to do so.

A solution is to block these merge commits from ever being pushed to the server. This is easily achieved using a git hook on the server side. The file 'update.merge' in the git-module repository gives an example of how to so. Place this file in the hooks directory of the repository on your server, rename it to 'update' and make sure it is executable. It will then block these annoying merge commits while allowing through genuine merge commits from named branches.

= Super repository =
With repository mirrors created and all modules now available as git repositories, it is quite convenient to have a container repository to manage everything. This container can be implemented as a git super-repository with git submodules for each component. In this section, we describe how to setup and administer such a repository, both locally and using a central server.

== Creating a super repository ==

== Common tasks ==

=== Adding a submodule ===
Adding a new submodule is as simple as running:

git module add <url-to-repository> <path-to-submodule> [<upstream-url>] [<upstream-type>] [<track-branch>]

checking the changes look OK, committing and then pushing. The optional arguments <upstream-url>, <upstream-type> and <upstream-branch> specify the URL and type of the upstream version of this repository and the branch which the super-repository should track. If not given, <upstream-url> is assumed to be the same as <url-to-repository> and <upstream-type> is assumed to be 'git'. Other possible options for <upstream-type> are 'hg' and 'svn'. If <track-branch> is not given it defaults to 'trunk' for git-svn repositories and 'master' for git and hg-git repositories.

=== Removing a submodule ===
To remove a submodule from the super repository, use:

git module rm <submodule-path>

check the changes look OK, then commit and push.

''Include here a guide to setting up a Git server with a Cactus super-repository''

Have temporarily copy+pasted the README which is in note form - need to write a proper up-to-date guide.

<pre>

###############################################################################
# CactusGit
###############################################################################

# Introduction

# This package allows you to check out Cactus using GetComponents from
# a CRL thornlist using git-svn instead of svn for the svn
# repositories. This means that your local Cactus tree contains
# complete version information and history and can be managed using
# git rather than svn.

# Further, the package provides tools for creating a
# "super-repository" for the tree with pointers to the downloaded
# repositories as git submodules. This means that the state of the
# tree can be identified by a single commit ID and can be cloned
# directly without having to use GetComponents or git-svn.

# Notes
# * To make this more convenient, small wrapper scripts can be written
# to replace the recipes given below
#
# * GetComponents could be extended to support git-svn directly,
# rather than using the wrapper-script approach that is currently
# used

# Tutorial

# Add our tools to the path, including the svn wrapper which tricks
# GetComponents into making a git-svn clone of each svn repository.
# We also include a copy of GetComponents here for convenience.
export OLDPATH=$PATH PATH=$PWD/bin:$PATH

# Check out the tree into the Cactus directory (can take a very long
# time, ~60 minutes, and does not show progress for each repository
# until it has checked it out completely)
GetComponents -v -a WaveToy.th

# Now manually checkout any thorns which require authentication
git svn clone ...

# Add the git-svn repos to your server (also add the Cactus super-repo).
crlrepos WaveToy.th > gitsvn.list
cd <gitolite_dir>
vi conf/gitolite.conf # Add the repositories in gitsvn.list
git commit conf/gitolite.conf
git push

# Add a 'fetching and pushing' (fp) user
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

# Push as the fp user
scp Cactus fp@server:fetch/
scp gitsvn.list fp@server:fetch/
ssh fp@server
cd fetch
for i in `cat gitsvn.list`; do cd $i && git remote add bare cactus@localhost:$i && git config --unset remote.bare.fetch && git config remote.bare.push 'refs/remotes/*:refs/heads/*' && git push bare && cd -; done

# On the server, update the HEAD of the git-svn repos
ssh cactus@server
cd ~/repositories
for i in `cat ../fetch/gitsvn`; do cd $i.git && git symbolic-ref HEAD refs/heads/git-svn && cd -; done

# Generate a super-repository called CactusGit using all the
# repositories from Cactus listed in WaveToy.th
crltogit CactusGit cactus@server: Llama.CTGamma.ET.th

# Add non-svn repos
GetComponents Llama.CTGamma.ET.th
rm -rf repos
git submodule add ... repos/...

# Symlink flesh
ln -s flesh/CONTRIBUTORS
ln -s flesh/COPYRIGHT
ln -s flesh/Makefile
ln -s flesh/doc
ln -s flesh/lib

# Commit symlinks (as fp)
git add *
git commit ...
git push cactus@localhost:Cactus master:master

######### Administrators ########
# Updating the git-svn mirrors
for i in `cat gitsvn.list`; do cd $i; git svn fetch && git push bare; cd -; done

# Updating the hg-git mirrors
for i in `cat githg.list`; do cd $i; hg pull && hg push git; cd -; done

# Updating the git mirrors
for i in `cat git.list`; do cd $i; git fetch -a -u && git push bare --all; cd -; done

# Updating all submodules in the super-repo
cd Cactus
git pull
git submodule foreach git pull
git commit -m "Update submodules." -a

# Updating a single submodule in the super-repo
cd Cactus
cd <repo-path>
git pull
cd -
git commit -m "Update submodule <repo-path>." <repo-path>

# Automatic updates of super-repo
git clone --recursive cactus@localhost:Cactus
cd Cactus
git submodule foreach 'git remote set-url origin /var/cactus/repositories/$path cactus@git.barrywardell.net:/$path; echo OK'
git remote set-url origin /var/cactus/repositories/Cactus
git remote set-url --push origin cactus@git.barrywardell.net:Cactus
git checkout submodules
git pull origin submodules
git submodule foreach 'git fetch origin && git checkout origin'
git commit -m "Update submodules." -a
git push origin submodules

# Adding a git-svn repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi gitsvn.list # add <newrepo>
git svn clone -s <url> <newrepo>
cd <newrepo>
git remote add bare cactus@localhost:<newrepo>
git config --unset remote.bare.fetch
git config remote.bare.push 'refs/remotes/*:refs/heads/*'
git push bare
cd ~/repositories/<newrepo>
git symbolic-ref HEAD refs/heads/trunk
cd ~/fetch/<newrepo>
git pull bare

# Adding a hg-git repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi githg.list # add <newrepo>
hg clone <url> <newrepo>
cd <newrepo>
vi .hg/hgrc # Add:
# [path]
# git = git+ssh://localhost/<newrepo>
# [git]
# intree = 1
hg gexport # Can take quite a long time
hg push git

# Adding a git mirror repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi git.list # add <newrepo>
git clone --mirror <url> <newrepo>
cd <newrepo>
git remote add bare cactus@localhost:<newrepo>
git config --unset remote.bare.fetch
git push bare --all

# Adding a submodule to the super-repo
git submodule add <url> <path>
git commit
git push

# Removing a submodule from the super-repo
vi .gitmodules
vi .git/config
git rm --cached <path>
git commit ...
git push

# Changing svn URL for a git-svn clone. WARNING: this is rewriting the history of
# the git-svn branch of your mirror and will mess peoples checkout up.
cd fetch/<repo>
git status
git gc
git filter-branch --msg-filter 'sed "s/git-svn-id: http/git-svn-id: https/g"' $(cat .git/packed-refs | awk '// {print $2}' | grep -v 'pack-refs')
sed -i -e "s/http/https/g" .git/config
rm -rf .git/svn
git svn rebase
git push bare --force

# Changing svn URL without rewriting history (https://git.wiki.kernel.org/index.php/GitSvnSwitch):
Edit the svn-remote url URL in .git/config to point to the new domain name
Run git svn fetch - This needs to fetch at least one new revision from svn!
Change svn-remote url back to the original url
Run git svn rebase -l to do a local rebase (with the changes that came in with the last fetch operation)
Change svn-remote url back to the new url
Run git svn rebase should now work again!

</pre>

GitSuperRepoAdminGuide

2011-07-12T10:23:45Z

Barry.wardell: /* Setting up Gitolite */

[[Category:GitSuperRepo]]

= Local mirrors =
The modules in a Cactus tree typically constitute a large number of repositories in various version control systems from wide number of sources. While GetComponents makes it get everything from their original locations, it may be still desirable to create local mirrors of these modules for several reasons:
* '''Speed''' - Some sources may be on the other side of the world so having a nearby local mirror can dramatically reduce the time to access them.
* '''Privacy''' - You may want to develop a new feature privately within your group before publicly releasing it. If this feature is based on an existing module then having a local mirror means you can create local private development branch which can later be easily merged and pushed upstream.
* '''Convenience''' - With remote repositories in a variety of version control systems, it is easy to get confused. Having local mirrors of these in a single version control system (i.e. git) simplifies things so that only a single tool is needed. In the case where the remote repository is in SVN local git mirrors also provides the advantage of offering the additional features git provides over svn.

This section describes how to setup, host and administer local mirrors of remote repositories.

== Creating repository mirrors ==

== Adding repository mirrors ==

== Removing repository mirrors ==

= Hosting mirrors on a server =

Rather than everyone creating their own local mirrors which they have to worry about keeping up-to-date, it makes sense to have a central server do so. In this section, we describe how to setup a server using [https://github.com/sitaramc/gitolite Gitolite].

== Setting up Gitolite ==
Follow the Gitolite [http://sitaramc.github.com/gitolite/doc/1-INSTALL.html#_root_method installation instructions]:

On the server (as root):
useradd -d /var/git -s /bin/bash -m git
apt-get install git-core
git clone git://github.com/sitaramc/gitolite
cd gitolite
src/gl-system-install
su - git
gl-setup /tmp/admin.pub

On your workstation:
git clone git@server:gitolite-admin

== Permissions ==

== Optional Extras ==
=== Blocking pushes of unnecessary merge commits ===

While working on a change in a git repository, it often happens that while you make local commits someone else pushes their own changes to the central repository. If you then try to push your changes, your push will be rejected as a non-fast-forward commit. The easiest solution is to pull the new changes from the server before pushing. However, this will introduce a 'merge commit' in the process. These merge commits rapidly build up and clutter up the history of the repository. For this reason, instead of pulling and introducing a merge commit you should do one of two things:
* Rebase your changes on top of the central repository. This can be achieved using 'git pull --rebase'.
* If your changes are significant, you can create a new named branch for them and merge that branch into the branch from the central repository.
In an ideal world, everyone would always follow these rules. However, in practice it is easy to forget to do so.

A solution is to block these merge commits from ever being pushed to the server. This is easily achieved using a git hook on the server side. The file 'update.merge' in the git-module repository gives an example of how to so. Place this file in the hooks directory of the repository on your server, rename it to 'update' and make sure it is executable. It will then block these annoying merge commits while allowing through genuine merge commits from named branches.

= Super repository =
With repository mirrors created and all modules now available as git repositories, it is quite convenient to have a container repository to manage everything. This container can be implemented as a git super-repository with git submodules for each component. In this section, we describe how to setup and administer such a repository, both locally and using a central server.

== Creating a super repository ==

== Common tasks ==

=== Adding a submodule ===
Adding a new submodule is as simple as running:

git module add <url-to-repository> <path-to-submodule> [<upstream-url>] [<upstream-type>] [<track-branch>]

checking the changes look OK, committing and then pushing. The optional arguments <upstream-url>, <upstream-type> and <upstream-branch> specify the URL and type of the upstream version of this repository and the branch which the super-repository should track. If not given, <upstream-url> is assumed to be the same as <url-to-repository> and <upstream-type> is assumed to be 'git'. Other possible options for <upstream-type> are 'hg' and 'svn'. If <track-branch> is not given it defaults to 'trunk' for git-svn repositories and 'master' for git and hg-git repositories.

=== Removing a submodule ===
To remove a submodule from the super repository, use:

git module rm <submodule-path>

check the changes look OK, then commit and push.

''Include here a guide to setting up a Git server with a Cactus super-repository''

Have temporarily copy+pasted the README which is in note form - need to write a proper up-to-date guide.

<pre>

###############################################################################
# CactusGit
###############################################################################

# Introduction

# This package allows you to check out Cactus using GetComponents from
# a CRL thornlist using git-svn instead of svn for the svn
# repositories. This means that your local Cactus tree contains
# complete version information and history and can be managed using
# git rather than svn.

# Further, the package provides tools for creating a
# "super-repository" for the tree with pointers to the downloaded
# repositories as git submodules. This means that the state of the
# tree can be identified by a single commit ID and can be cloned
# directly without having to use GetComponents or git-svn.

# Notes
# * To make this more convenient, small wrapper scripts can be written
# to replace the recipes given below
#
# * GetComponents could be extended to support git-svn directly,
# rather than using the wrapper-script approach that is currently
# used

# Tutorial

# Add our tools to the path, including the svn wrapper which tricks
# GetComponents into making a git-svn clone of each svn repository.
# We also include a copy of GetComponents here for convenience.
export OLDPATH=$PATH PATH=$PWD/bin:$PATH

# Check out the tree into the Cactus directory (can take a very long
# time, ~60 minutes, and does not show progress for each repository
# until it has checked it out completely)
GetComponents -v -a WaveToy.th

# Now manually checkout any thorns which require authentication
git svn clone ...

# Add the git-svn repos to your server (also add the Cactus super-repo).
crlrepos WaveToy.th > gitsvn.list
cd <gitolite_dir>
vi conf/gitolite.conf # Add the repositories in gitsvn.list
git commit conf/gitolite.conf
git push

# Add a 'fetching and pushing' (fp) user
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

# Push as the fp user
scp Cactus fp@server:fetch/
scp gitsvn.list fp@server:fetch/
ssh fp@server
cd fetch
for i in `cat gitsvn.list`; do cd $i && git remote add bare cactus@localhost:$i && git config --unset remote.bare.fetch && git config remote.bare.push 'refs/remotes/*:refs/heads/*' && git push bare && cd -; done

# On the server, update the HEAD of the git-svn repos
ssh cactus@server
cd ~/repositories
for i in `cat ../fetch/gitsvn`; do cd $i.git && git symbolic-ref HEAD refs/heads/git-svn && cd -; done

# Generate a super-repository called CactusGit using all the
# repositories from Cactus listed in WaveToy.th
crltogit CactusGit cactus@server: Llama.CTGamma.ET.th

# Add non-svn repos
GetComponents Llama.CTGamma.ET.th
rm -rf repos
git submodule add ... repos/...

# Symlink flesh
ln -s flesh/CONTRIBUTORS
ln -s flesh/COPYRIGHT
ln -s flesh/Makefile
ln -s flesh/doc
ln -s flesh/lib

# Commit symlinks (as fp)
git add *
git commit ...
git push cactus@localhost:Cactus master:master

######### Administrators ########
# Updating the git-svn mirrors
for i in `cat gitsvn.list`; do cd $i; git svn fetch && git push bare; cd -; done

# Updating the hg-git mirrors
for i in `cat githg.list`; do cd $i; hg pull && hg push git; cd -; done

# Updating the git mirrors
for i in `cat git.list`; do cd $i; git fetch -a -u && git push bare --all; cd -; done

# Updating all submodules in the super-repo
cd Cactus
git pull
git submodule foreach git pull
git commit -m "Update submodules." -a

# Updating a single submodule in the super-repo
cd Cactus
cd <repo-path>
git pull
cd -
git commit -m "Update submodule <repo-path>." <repo-path>

# Automatic updates of super-repo
git clone --recursive cactus@localhost:Cactus
cd Cactus
git submodule foreach 'git remote set-url origin /var/cactus/repositories/$path cactus@git.barrywardell.net:/$path; echo OK'
git remote set-url origin /var/cactus/repositories/Cactus
git remote set-url --push origin cactus@git.barrywardell.net:Cactus
git checkout submodules
git pull origin submodules
git submodule foreach 'git fetch origin && git checkout origin'
git commit -m "Update submodules." -a
git push origin submodules

# Adding a git-svn repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi gitsvn.list # add <newrepo>
git svn clone -s <url> <newrepo>
cd <newrepo>
git remote add bare cactus@localhost:<newrepo>
git config --unset remote.bare.fetch
git config remote.bare.push 'refs/remotes/*:refs/heads/*'
git push bare
cd ~/repositories/<newrepo>
git symbolic-ref HEAD refs/heads/trunk
cd ~/fetch/<newrepo>
git pull bare

# Adding a hg-git repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi githg.list # add <newrepo>
hg clone <url> <newrepo>
cd <newrepo>
vi .hg/hgrc # Add:
# [path]
# git = git+ssh://localhost/<newrepo>
# [git]
# intree = 1
hg gexport # Can take quite a long time
hg push git

# Adding a git mirror repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi git.list # add <newrepo>
git clone --mirror <url> <newrepo>
cd <newrepo>
git remote add bare cactus@localhost:<newrepo>
git config --unset remote.bare.fetch
git push bare --all

# Adding a submodule to the super-repo
git submodule add <url> <path>
git commit
git push

# Removing a submodule from the super-repo
vi .gitmodules
vi .git/config
git rm --cached <path>
git commit ...
git push

# Changing svn URL for a git-svn clone. WARNING: this is rewriting the history of
# the git-svn branch of your mirror and will mess peoples checkout up.
cd fetch/<repo>
git status
git gc
git filter-branch --msg-filter 'sed "s/git-svn-id: http/git-svn-id: https/g"' $(cat .git/packed-refs | awk '// {print $2}' | grep -v 'pack-refs')
sed -i -e "s/http/https/g" .git/config
rm -rf .git/svn
git svn rebase
git push bare --force

# Changing svn URL without rewriting history (https://git.wiki.kernel.org/index.php/GitSvnSwitch):
Edit the svn-remote url URL in .git/config to point to the new domain name
Run git svn fetch - This needs to fetch at least one new revision from svn!
Change svn-remote url back to the original url
Run git svn rebase -l to do a local rebase (with the changes that came in with the last fetch operation)
Change svn-remote url back to the new url
Run git svn rebase should now work again!

</pre>

GitSuperRepoAdminGuide

2011-07-12T10:22:26Z

Barry.wardell: /* Setting up Gitolite */

[[Category:GitSuperRepo]]

= Local mirrors =
The modules in a Cactus tree typically constitute a large number of repositories in various version control systems from wide number of sources. While GetComponents makes it get everything from their original locations, it may be still desirable to create local mirrors of these modules for several reasons:
* '''Speed''' - Some sources may be on the other side of the world so having a nearby local mirror can dramatically reduce the time to access them.
* '''Privacy''' - You may want to develop a new feature privately within your group before publicly releasing it. If this feature is based on an existing module then having a local mirror means you can create local private development branch which can later be easily merged and pushed upstream.
* '''Convenience''' - With remote repositories in a variety of version control systems, it is easy to get confused. Having local mirrors of these in a single version control system (i.e. git) simplifies things so that only a single tool is needed. In the case where the remote repository is in SVN local git mirrors also provides the advantage of offering the additional features git provides over svn.

This section describes how to setup, host and administer local mirrors of remote repositories.

== Creating repository mirrors ==

== Adding repository mirrors ==

== Removing repository mirrors ==

= Hosting mirrors on a server =

Rather than everyone creating their own local mirrors which they have to worry about keeping up-to-date, it makes sense to have a central server do so. In this section, we describe how to setup a server using [https://github.com/sitaramc/gitolite Gitolite].

== Setting up Gitolite ==
Follow the Gitolite [http://sitaramc.github.com/gitolite/doc/1-INSTALL.html#_root_method installation instructions]:
useradd -d /var/git -s /bin/bash -m git
apt-get install git-core
git clone git://github.com/sitaramc/gitolite
cd gitolite
src/gl-system-install
gl-setup /tmp/admin.pub
git clone git@server:gitolite-admin

== Permissions ==

== Optional Extras ==
=== Blocking pushes of unnecessary merge commits ===

While working on a change in a git repository, it often happens that while you make local commits someone else pushes their own changes to the central repository. If you then try to push your changes, your push will be rejected as a non-fast-forward commit. The easiest solution is to pull the new changes from the server before pushing. However, this will introduce a 'merge commit' in the process. These merge commits rapidly build up and clutter up the history of the repository. For this reason, instead of pulling and introducing a merge commit you should do one of two things:
* Rebase your changes on top of the central repository. This can be achieved using 'git pull --rebase'.
* If your changes are significant, you can create a new named branch for them and merge that branch into the branch from the central repository.
In an ideal world, everyone would always follow these rules. However, in practice it is easy to forget to do so.

A solution is to block these merge commits from ever being pushed to the server. This is easily achieved using a git hook on the server side. The file 'update.merge' in the git-module repository gives an example of how to so. Place this file in the hooks directory of the repository on your server, rename it to 'update' and make sure it is executable. It will then block these annoying merge commits while allowing through genuine merge commits from named branches.

= Super repository =
With repository mirrors created and all modules now available as git repositories, it is quite convenient to have a container repository to manage everything. This container can be implemented as a git super-repository with git submodules for each component. In this section, we describe how to setup and administer such a repository, both locally and using a central server.

== Creating a super repository ==

== Common tasks ==

=== Adding a submodule ===
Adding a new submodule is as simple as running:

git module add <url-to-repository> <path-to-submodule> [<upstream-url>] [<upstream-type>] [<track-branch>]

checking the changes look OK, committing and then pushing. The optional arguments <upstream-url>, <upstream-type> and <upstream-branch> specify the URL and type of the upstream version of this repository and the branch which the super-repository should track. If not given, <upstream-url> is assumed to be the same as <url-to-repository> and <upstream-type> is assumed to be 'git'. Other possible options for <upstream-type> are 'hg' and 'svn'. If <track-branch> is not given it defaults to 'trunk' for git-svn repositories and 'master' for git and hg-git repositories.

=== Removing a submodule ===
To remove a submodule from the super repository, use:

git module rm <submodule-path>

check the changes look OK, then commit and push.

''Include here a guide to setting up a Git server with a Cactus super-repository''

Have temporarily copy+pasted the README which is in note form - need to write a proper up-to-date guide.

<pre>

###############################################################################
# CactusGit
###############################################################################

# Introduction

# This package allows you to check out Cactus using GetComponents from
# a CRL thornlist using git-svn instead of svn for the svn
# repositories. This means that your local Cactus tree contains
# complete version information and history and can be managed using
# git rather than svn.

# Further, the package provides tools for creating a
# "super-repository" for the tree with pointers to the downloaded
# repositories as git submodules. This means that the state of the
# tree can be identified by a single commit ID and can be cloned
# directly without having to use GetComponents or git-svn.

# Notes
# * To make this more convenient, small wrapper scripts can be written
# to replace the recipes given below
#
# * GetComponents could be extended to support git-svn directly,
# rather than using the wrapper-script approach that is currently
# used

# Tutorial

# Add our tools to the path, including the svn wrapper which tricks
# GetComponents into making a git-svn clone of each svn repository.
# We also include a copy of GetComponents here for convenience.
export OLDPATH=$PATH PATH=$PWD/bin:$PATH

# Check out the tree into the Cactus directory (can take a very long
# time, ~60 minutes, and does not show progress for each repository
# until it has checked it out completely)
GetComponents -v -a WaveToy.th

# Now manually checkout any thorns which require authentication
git svn clone ...

# Add the git-svn repos to your server (also add the Cactus super-repo).
crlrepos WaveToy.th > gitsvn.list
cd <gitolite_dir>
vi conf/gitolite.conf # Add the repositories in gitsvn.list
git commit conf/gitolite.conf
git push

# Add a 'fetching and pushing' (fp) user
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

# Push as the fp user
scp Cactus fp@server:fetch/
scp gitsvn.list fp@server:fetch/
ssh fp@server
cd fetch
for i in `cat gitsvn.list`; do cd $i && git remote add bare cactus@localhost:$i && git config --unset remote.bare.fetch && git config remote.bare.push 'refs/remotes/*:refs/heads/*' && git push bare && cd -; done

# On the server, update the HEAD of the git-svn repos
ssh cactus@server
cd ~/repositories
for i in `cat ../fetch/gitsvn`; do cd $i.git && git symbolic-ref HEAD refs/heads/git-svn && cd -; done

# Generate a super-repository called CactusGit using all the
# repositories from Cactus listed in WaveToy.th
crltogit CactusGit cactus@server: Llama.CTGamma.ET.th

# Add non-svn repos
GetComponents Llama.CTGamma.ET.th
rm -rf repos
git submodule add ... repos/...

# Symlink flesh
ln -s flesh/CONTRIBUTORS
ln -s flesh/COPYRIGHT
ln -s flesh/Makefile
ln -s flesh/doc
ln -s flesh/lib

# Commit symlinks (as fp)
git add *
git commit ...
git push cactus@localhost:Cactus master:master

######### Administrators ########
# Updating the git-svn mirrors
for i in `cat gitsvn.list`; do cd $i; git svn fetch && git push bare; cd -; done

# Updating the hg-git mirrors
for i in `cat githg.list`; do cd $i; hg pull && hg push git; cd -; done

# Updating the git mirrors
for i in `cat git.list`; do cd $i; git fetch -a -u && git push bare --all; cd -; done

# Updating all submodules in the super-repo
cd Cactus
git pull
git submodule foreach git pull
git commit -m "Update submodules." -a

# Updating a single submodule in the super-repo
cd Cactus
cd <repo-path>
git pull
cd -
git commit -m "Update submodule <repo-path>." <repo-path>

# Automatic updates of super-repo
git clone --recursive cactus@localhost:Cactus
cd Cactus
git submodule foreach 'git remote set-url origin /var/cactus/repositories/$path cactus@git.barrywardell.net:/$path; echo OK'
git remote set-url origin /var/cactus/repositories/Cactus
git remote set-url --push origin cactus@git.barrywardell.net:Cactus
git checkout submodules
git pull origin submodules
git submodule foreach 'git fetch origin && git checkout origin'
git commit -m "Update submodules." -a
git push origin submodules

# Adding a git-svn repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi gitsvn.list # add <newrepo>
git svn clone -s <url> <newrepo>
cd <newrepo>
git remote add bare cactus@localhost:<newrepo>
git config --unset remote.bare.fetch
git config remote.bare.push 'refs/remotes/*:refs/heads/*'
git push bare
cd ~/repositories/<newrepo>
git symbolic-ref HEAD refs/heads/trunk
cd ~/fetch/<newrepo>
git pull bare

# Adding a hg-git repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi githg.list # add <newrepo>
hg clone <url> <newrepo>
cd <newrepo>
vi .hg/hgrc # Add:
# [path]
# git = git+ssh://localhost/<newrepo>
# [git]
# intree = 1
hg gexport # Can take quite a long time
hg push git

# Adding a git mirror repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi git.list # add <newrepo>
git clone --mirror <url> <newrepo>
cd <newrepo>
git remote add bare cactus@localhost:<newrepo>
git config --unset remote.bare.fetch
git push bare --all

# Adding a submodule to the super-repo
git submodule add <url> <path>
git commit
git push

# Removing a submodule from the super-repo
vi .gitmodules
vi .git/config
git rm --cached <path>
git commit ...
git push

# Changing svn URL for a git-svn clone. WARNING: this is rewriting the history of
# the git-svn branch of your mirror and will mess peoples checkout up.
cd fetch/<repo>
git status
git gc
git filter-branch --msg-filter 'sed "s/git-svn-id: http/git-svn-id: https/g"' $(cat .git/packed-refs | awk '// {print $2}' | grep -v 'pack-refs')
sed -i -e "s/http/https/g" .git/config
rm -rf .git/svn
git svn rebase
git push bare --force

# Changing svn URL without rewriting history (https://git.wiki.kernel.org/index.php/GitSvnSwitch):
Edit the svn-remote url URL in .git/config to point to the new domain name
Run git svn fetch - This needs to fetch at least one new revision from svn!
Change svn-remote url back to the original url
Run git svn rebase -l to do a local rebase (with the changes that came in with the last fetch operation)
Change svn-remote url back to the new url
Run git svn rebase should now work again!

</pre>

GitSuperRepoAdminGuide

2011-07-12T10:22:02Z

Barry.wardell: /* Hosting mirrors on a server */

[[Category:GitSuperRepo]]

= Local mirrors =
The modules in a Cactus tree typically constitute a large number of repositories in various version control systems from wide number of sources. While GetComponents makes it get everything from their original locations, it may be still desirable to create local mirrors of these modules for several reasons:
* '''Speed''' - Some sources may be on the other side of the world so having a nearby local mirror can dramatically reduce the time to access them.
* '''Privacy''' - You may want to develop a new feature privately within your group before publicly releasing it. If this feature is based on an existing module then having a local mirror means you can create local private development branch which can later be easily merged and pushed upstream.
* '''Convenience''' - With remote repositories in a variety of version control systems, it is easy to get confused. Having local mirrors of these in a single version control system (i.e. git) simplifies things so that only a single tool is needed. In the case where the remote repository is in SVN local git mirrors also provides the advantage of offering the additional features git provides over svn.

This section describes how to setup, host and administer local mirrors of remote repositories.

== Creating repository mirrors ==

== Adding repository mirrors ==

== Removing repository mirrors ==

= Hosting mirrors on a server =

Rather than everyone creating their own local mirrors which they have to worry about keeping up-to-date, it makes sense to have a central server do so. In this section, we describe how to setup a server using [https://github.com/sitaramc/gitolite Gitolite].

== Setting up Gitolite ==
Follow the Gitolite [http://sitaramc.github.com/gitolite/doc/1-INSTALL.html#_root_method installation instructions]:
useradd -d /var/git -s /bin/bash -m git
apt-get install git-core
git clone git://github.com/sitaramc/gitolite
cd gitolite
src/gl-system-install
gl-setup /tmp/admin.pub
git clone git@server:gitolite-admin

== Permissions ==

== Optional Extras ==
=== Blocking pushes of unnecessary merge commits ===

While working on a change in a git repository, it often happens that while you make local commits someone else pushes their own changes to the central repository. If you then try to push your changes, your push will be rejected as a non-fast-forward commit. The easiest solution is to pull the new changes from the server before pushing. However, this will introduce a 'merge commit' in the process. These merge commits rapidly build up and clutter up the history of the repository. For this reason, instead of pulling and introducing a merge commit you should do one of two things:
* Rebase your changes on top of the central repository. This can be achieved using 'git pull --rebase'.
* If your changes are significant, you can create a new named branch for them and merge that branch into the branch from the central repository.
In an ideal world, everyone would always follow these rules. However, in practice it is easy to forget to do so.

A solution is to block these merge commits from ever being pushed to the server. This is easily achieved using a git hook on the server side. The file 'update.merge' in the git-module repository gives an example of how to so. Place this file in the hooks directory of the repository on your server, rename it to 'update' and make sure it is executable. It will then block these annoying merge commits while allowing through genuine merge commits from named branches.

= Super repository =
With repository mirrors created and all modules now available as git repositories, it is quite convenient to have a container repository to manage everything. This container can be implemented as a git super-repository with git submodules for each component. In this section, we describe how to setup and administer such a repository, both locally and using a central server.

== Creating a super repository ==

== Common tasks ==

=== Adding a submodule ===
Adding a new submodule is as simple as running:

git module add <url-to-repository> <path-to-submodule> [<upstream-url>] [<upstream-type>] [<track-branch>]

checking the changes look OK, committing and then pushing. The optional arguments <upstream-url>, <upstream-type> and <upstream-branch> specify the URL and type of the upstream version of this repository and the branch which the super-repository should track. If not given, <upstream-url> is assumed to be the same as <url-to-repository> and <upstream-type> is assumed to be 'git'. Other possible options for <upstream-type> are 'hg' and 'svn'. If <track-branch> is not given it defaults to 'trunk' for git-svn repositories and 'master' for git and hg-git repositories.

=== Removing a submodule ===
To remove a submodule from the super repository, use:

git module rm <submodule-path>

check the changes look OK, then commit and push.

''Include here a guide to setting up a Git server with a Cactus super-repository''

Have temporarily copy+pasted the README which is in note form - need to write a proper up-to-date guide.

<pre>

###############################################################################
# CactusGit
###############################################################################

# Introduction

# This package allows you to check out Cactus using GetComponents from
# a CRL thornlist using git-svn instead of svn for the svn
# repositories. This means that your local Cactus tree contains
# complete version information and history and can be managed using
# git rather than svn.

# Further, the package provides tools for creating a
# "super-repository" for the tree with pointers to the downloaded
# repositories as git submodules. This means that the state of the
# tree can be identified by a single commit ID and can be cloned
# directly without having to use GetComponents or git-svn.

# Notes
# * To make this more convenient, small wrapper scripts can be written
# to replace the recipes given below
#
# * GetComponents could be extended to support git-svn directly,
# rather than using the wrapper-script approach that is currently
# used

# Tutorial

# Add our tools to the path, including the svn wrapper which tricks
# GetComponents into making a git-svn clone of each svn repository.
# We also include a copy of GetComponents here for convenience.
export OLDPATH=$PATH PATH=$PWD/bin:$PATH

# Check out the tree into the Cactus directory (can take a very long
# time, ~60 minutes, and does not show progress for each repository
# until it has checked it out completely)
GetComponents -v -a WaveToy.th

# Now manually checkout any thorns which require authentication
git svn clone ...

# Add the git-svn repos to your server (also add the Cactus super-repo).
crlrepos WaveToy.th > gitsvn.list
cd <gitolite_dir>
vi conf/gitolite.conf # Add the repositories in gitsvn.list
git commit conf/gitolite.conf
git push

# Add a 'fetching and pushing' (fp) user
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

# Push as the fp user
scp Cactus fp@server:fetch/
scp gitsvn.list fp@server:fetch/
ssh fp@server
cd fetch
for i in `cat gitsvn.list`; do cd $i && git remote add bare cactus@localhost:$i && git config --unset remote.bare.fetch && git config remote.bare.push 'refs/remotes/*:refs/heads/*' && git push bare && cd -; done

# On the server, update the HEAD of the git-svn repos
ssh cactus@server
cd ~/repositories
for i in `cat ../fetch/gitsvn`; do cd $i.git && git symbolic-ref HEAD refs/heads/git-svn && cd -; done

# Generate a super-repository called CactusGit using all the
# repositories from Cactus listed in WaveToy.th
crltogit CactusGit cactus@server: Llama.CTGamma.ET.th

# Add non-svn repos
GetComponents Llama.CTGamma.ET.th
rm -rf repos
git submodule add ... repos/...

# Symlink flesh
ln -s flesh/CONTRIBUTORS
ln -s flesh/COPYRIGHT
ln -s flesh/Makefile
ln -s flesh/doc
ln -s flesh/lib

# Commit symlinks (as fp)
git add *
git commit ...
git push cactus@localhost:Cactus master:master

######### Administrators ########
# Updating the git-svn mirrors
for i in `cat gitsvn.list`; do cd $i; git svn fetch && git push bare; cd -; done

# Updating the hg-git mirrors
for i in `cat githg.list`; do cd $i; hg pull && hg push git; cd -; done

# Updating the git mirrors
for i in `cat git.list`; do cd $i; git fetch -a -u && git push bare --all; cd -; done

# Updating all submodules in the super-repo
cd Cactus
git pull
git submodule foreach git pull
git commit -m "Update submodules." -a

# Updating a single submodule in the super-repo
cd Cactus
cd <repo-path>
git pull
cd -
git commit -m "Update submodule <repo-path>." <repo-path>

# Automatic updates of super-repo
git clone --recursive cactus@localhost:Cactus
cd Cactus
git submodule foreach 'git remote set-url origin /var/cactus/repositories/$path cactus@git.barrywardell.net:/$path; echo OK'
git remote set-url origin /var/cactus/repositories/Cactus
git remote set-url --push origin cactus@git.barrywardell.net:Cactus
git checkout submodules
git pull origin submodules
git submodule foreach 'git fetch origin && git checkout origin'
git commit -m "Update submodules." -a
git push origin submodules

# Adding a git-svn repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi gitsvn.list # add <newrepo>
git svn clone -s <url> <newrepo>
cd <newrepo>
git remote add bare cactus@localhost:<newrepo>
git config --unset remote.bare.fetch
git config remote.bare.push 'refs/remotes/*:refs/heads/*'
git push bare
cd ~/repositories/<newrepo>
git symbolic-ref HEAD refs/heads/trunk
cd ~/fetch/<newrepo>
git pull bare

# Adding a hg-git repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi githg.list # add <newrepo>
hg clone <url> <newrepo>
cd <newrepo>
vi .hg/hgrc # Add:
# [path]
# git = git+ssh://localhost/<newrepo>
# [git]
# intree = 1
hg gexport # Can take quite a long time
hg push git

# Adding a git mirror repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi git.list # add <newrepo>
git clone --mirror <url> <newrepo>
cd <newrepo>
git remote add bare cactus@localhost:<newrepo>
git config --unset remote.bare.fetch
git push bare --all

# Adding a submodule to the super-repo
git submodule add <url> <path>
git commit
git push

# Removing a submodule from the super-repo
vi .gitmodules
vi .git/config
git rm --cached <path>
git commit ...
git push

# Changing svn URL for a git-svn clone. WARNING: this is rewriting the history of
# the git-svn branch of your mirror and will mess peoples checkout up.
cd fetch/<repo>
git status
git gc
git filter-branch --msg-filter 'sed "s/git-svn-id: http/git-svn-id: https/g"' $(cat .git/packed-refs | awk '// {print $2}' | grep -v 'pack-refs')
sed -i -e "s/http/https/g" .git/config
rm -rf .git/svn
git svn rebase
git push bare --force

# Changing svn URL without rewriting history (https://git.wiki.kernel.org/index.php/GitSvnSwitch):
Edit the svn-remote url URL in .git/config to point to the new domain name
Run git svn fetch - This needs to fetch at least one new revision from svn!
Change svn-remote url back to the original url
Run git svn rebase -l to do a local rebase (with the changes that came in with the last fetch operation)
Change svn-remote url back to the new url
Run git svn rebase should now work again!

</pre>

GitSuperRepoAdminGuide

2011-06-27T16:08:22Z

Barry.wardell: /* Creating a repository */

[[Category:GitSuperRepo]]

= Local mirrors =
The modules in a Cactus tree typically constitute a large number of repositories in various version control systems from wide number of sources. While GetComponents makes it get everything from their original locations, it may be still desirable to create local mirrors of these modules for several reasons:
* '''Speed''' - Some sources may be on the other side of the world so having a nearby local mirror can dramatically reduce the time to access them.
* '''Privacy''' - You may want to develop a new feature privately within your group before publicly releasing it. If this feature is based on an existing module then having a local mirror means you can create local private development branch which can later be easily merged and pushed upstream.
* '''Convenience''' - With remote repositories in a variety of version control systems, it is easy to get confused. Having local mirrors of these in a single version control system (i.e. git) simplifies things so that only a single tool is needed. In the case where the remote repository is in SVN local git mirrors also provides the advantage of offering the additional features git provides over svn.

This section describes how to setup, host and administer local mirrors of remote repositories.

== Creating repository mirrors ==

== Adding repository mirrors ==

== Removing repository mirrors ==

= Hosting mirrors on a server =

Rather than everyone creating their own local mirrors which they have to worry about keeping up-to-date, it makes sense to have a central server do so. In this section, we describe how to setup a server using [https://github.com/sitaramc/gitolite Gitolite].

== Permissions ==

== Optional Extras ==
=== Blocking pushes of unnecessary merge commits ===

While working on a change in a git repository, it often happens that while you make local commits someone else pushes their own changes to the central repository. If you then try to push your changes, your push will be rejected as a non-fast-forward commit. The easiest solution is to pull the new changes from the server before pushing. However, this will introduce a 'merge commit' in the process. These merge commits rapidly build up and clutter up the history of the repository. For this reason, instead of pulling and introducing a merge commit you should do one of two things:
* Rebase your changes on top of the central repository. This can be achieved using 'git pull --rebase'.
* If your changes are significant, you can create a new named branch for them and merge that branch into the branch from the central repository.
In an ideal world, everyone would always follow these rules. However, in practice it is easy to forget to do so.

A solution is to block these merge commits from ever being pushed to the server. This is easily achieved using a git hook on the server side. The file 'update.merge' in the git-module repository gives an example of how to so. Place this file in the hooks directory of the repository on your server, rename it to 'update' and make sure it is executable. It will then block these annoying merge commits while allowing through genuine merge commits from named branches.

= Super repository =
With repository mirrors created and all modules now available as git repositories, it is quite convenient to have a container repository to manage everything. This container can be implemented as a git super-repository with git submodules for each component. In this section, we describe how to setup and administer such a repository, both locally and using a central server.

== Creating a super repository ==

== Common tasks ==

=== Adding a submodule ===
Adding a new submodule is as simple as running:

git module add <url-to-repository> <path-to-submodule> [<upstream-url>] [<upstream-type>] [<track-branch>]

checking the changes look OK, committing and then pushing. The optional arguments <upstream-url>, <upstream-type> and <upstream-branch> specify the URL and type of the upstream version of this repository and the branch which the super-repository should track. If not given, <upstream-url> is assumed to be the same as <url-to-repository> and <upstream-type> is assumed to be 'git'. Other possible options for <upstream-type> are 'hg' and 'svn'. If <track-branch> is not given it defaults to 'trunk' for git-svn repositories and 'master' for git and hg-git repositories.

=== Removing a submodule ===
To remove a submodule from the super repository, use:

git module rm <submodule-path>

check the changes look OK, then commit and push.

''Include here a guide to setting up a Git server with a Cactus super-repository''

Have temporarily copy+pasted the README which is in note form - need to write a proper up-to-date guide.

<pre>

###############################################################################
# CactusGit
###############################################################################

# Introduction

# This package allows you to check out Cactus using GetComponents from
# a CRL thornlist using git-svn instead of svn for the svn
# repositories. This means that your local Cactus tree contains
# complete version information and history and can be managed using
# git rather than svn.

# Further, the package provides tools for creating a
# "super-repository" for the tree with pointers to the downloaded
# repositories as git submodules. This means that the state of the
# tree can be identified by a single commit ID and can be cloned
# directly without having to use GetComponents or git-svn.

# Notes
# * To make this more convenient, small wrapper scripts can be written
# to replace the recipes given below
#
# * GetComponents could be extended to support git-svn directly,
# rather than using the wrapper-script approach that is currently
# used

# Tutorial

# Add our tools to the path, including the svn wrapper which tricks
# GetComponents into making a git-svn clone of each svn repository.
# We also include a copy of GetComponents here for convenience.
export OLDPATH=$PATH PATH=$PWD/bin:$PATH

# Check out the tree into the Cactus directory (can take a very long
# time, ~60 minutes, and does not show progress for each repository
# until it has checked it out completely)
GetComponents -v -a WaveToy.th

# Now manually checkout any thorns which require authentication
git svn clone ...

# Add the git-svn repos to your server (also add the Cactus super-repo).
crlrepos WaveToy.th > gitsvn.list
cd <gitolite_dir>
vi conf/gitolite.conf # Add the repositories in gitsvn.list
git commit conf/gitolite.conf
git push

# Add a 'fetching and pushing' (fp) user
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

# Push as the fp user
scp Cactus fp@server:fetch/
scp gitsvn.list fp@server:fetch/
ssh fp@server
cd fetch
for i in `cat gitsvn.list`; do cd $i && git remote add bare cactus@localhost:$i && git config --unset remote.bare.fetch && git config remote.bare.push 'refs/remotes/*:refs/heads/*' && git push bare && cd -; done

# On the server, update the HEAD of the git-svn repos
ssh cactus@server
cd ~/repositories
for i in `cat ../fetch/gitsvn`; do cd $i.git && git symbolic-ref HEAD refs/heads/git-svn && cd -; done

# Generate a super-repository called CactusGit using all the
# repositories from Cactus listed in WaveToy.th
crltogit CactusGit cactus@server: Llama.CTGamma.ET.th

# Add non-svn repos
GetComponents Llama.CTGamma.ET.th
rm -rf repos
git submodule add ... repos/...

# Symlink flesh
ln -s flesh/CONTRIBUTORS
ln -s flesh/COPYRIGHT
ln -s flesh/Makefile
ln -s flesh/doc
ln -s flesh/lib

# Commit symlinks (as fp)
git add *
git commit ...
git push cactus@localhost:Cactus master:master

######### Administrators ########
# Updating the git-svn mirrors
for i in `cat gitsvn.list`; do cd $i; git svn fetch && git push bare; cd -; done

# Updating the hg-git mirrors
for i in `cat githg.list`; do cd $i; hg pull && hg push git; cd -; done

# Updating the git mirrors
for i in `cat git.list`; do cd $i; git fetch -a -u && git push bare --all; cd -; done

# Updating all submodules in the super-repo
cd Cactus
git pull
git submodule foreach git pull
git commit -m "Update submodules." -a

# Updating a single submodule in the super-repo
cd Cactus
cd <repo-path>
git pull
cd -
git commit -m "Update submodule <repo-path>." <repo-path>

# Automatic updates of super-repo
git clone --recursive cactus@localhost:Cactus
cd Cactus
git submodule foreach 'git remote set-url origin /var/cactus/repositories/$path cactus@git.barrywardell.net:/$path; echo OK'
git remote set-url origin /var/cactus/repositories/Cactus
git remote set-url --push origin cactus@git.barrywardell.net:Cactus
git checkout submodules
git pull origin submodules
git submodule foreach 'git fetch origin && git checkout origin'
git commit -m "Update submodules." -a
git push origin submodules

# Adding a git-svn repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi gitsvn.list # add <newrepo>
git svn clone -s <url> <newrepo>
cd <newrepo>
git remote add bare cactus@localhost:<newrepo>
git config --unset remote.bare.fetch
git config remote.bare.push 'refs/remotes/*:refs/heads/*'
git push bare
cd ~/repositories/<newrepo>
git symbolic-ref HEAD refs/heads/trunk
cd ~/fetch/<newrepo>
git pull bare

# Adding a hg-git repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi githg.list # add <newrepo>
hg clone <url> <newrepo>
cd <newrepo>
vi .hg/hgrc # Add:
# [path]
# git = git+ssh://localhost/<newrepo>
# [git]
# intree = 1
hg gexport # Can take quite a long time
hg push git

# Adding a git mirror repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi git.list # add <newrepo>
git clone --mirror <url> <newrepo>
cd <newrepo>
git remote add bare cactus@localhost:<newrepo>
git config --unset remote.bare.fetch
git push bare --all

# Adding a submodule to the super-repo
git submodule add <url> <path>
git commit
git push

# Removing a submodule from the super-repo
vi .gitmodules
vi .git/config
git rm --cached <path>
git commit ...
git push

# Changing svn URL for a git-svn clone. WARNING: this is rewriting the history of
# the git-svn branch of your mirror and will mess peoples checkout up.
cd fetch/<repo>
git status
git gc
git filter-branch --msg-filter 'sed "s/git-svn-id: http/git-svn-id: https/g"' $(cat .git/packed-refs | awk '// {print $2}' | grep -v 'pack-refs')
sed -i -e "s/http/https/g" .git/config
rm -rf .git/svn
git svn rebase
git push bare --force

# Changing svn URL without rewriting history (https://git.wiki.kernel.org/index.php/GitSvnSwitch):
Edit the svn-remote url URL in .git/config to point to the new domain name
Run git svn fetch - This needs to fetch at least one new revision from svn!
Change svn-remote url back to the original url
Run git svn rebase -l to do a local rebase (with the changes that came in with the last fetch operation)
Change svn-remote url back to the new url
Run git svn rebase should now work again!

</pre>

GitSuperRepoAdminGuide

2011-06-27T16:05:26Z

Barry.wardell:

[[Category:GitSuperRepo]]

= Local mirrors =
The modules in a Cactus tree typically constitute a large number of repositories in various version control systems from wide number of sources. While GetComponents makes it get everything from their original locations, it may be still desirable to create local mirrors of these modules for several reasons:
* '''Speed''' - Some sources may be on the other side of the world so having a nearby local mirror can dramatically reduce the time to access them.
* '''Privacy''' - You may want to develop a new feature privately within your group before publicly releasing it. If this feature is based on an existing module then having a local mirror means you can create local private development branch which can later be easily merged and pushed upstream.
* '''Convenience''' - With remote repositories in a variety of version control systems, it is easy to get confused. Having local mirrors of these in a single version control system (i.e. git) simplifies things so that only a single tool is needed. In the case where the remote repository is in SVN local git mirrors also provides the advantage of offering the additional features git provides over svn.

This section describes how to setup, host and administer local mirrors of remote repositories.

== Creating repository mirrors ==

== Adding repository mirrors ==

== Removing repository mirrors ==

= Hosting mirrors on a server =

Rather than everyone creating their own local mirrors which they have to worry about keeping up-to-date, it makes sense to have a central server do so. In this section, we describe how to setup a server using [https://github.com/sitaramc/gitolite Gitolite].

== Permissions ==

== Optional Extras ==
=== Blocking pushes of unnecessary merge commits ===

While working on a change in a git repository, it often happens that while you make local commits someone else pushes their own changes to the central repository. If you then try to push your changes, your push will be rejected as a non-fast-forward commit. The easiest solution is to pull the new changes from the server before pushing. However, this will introduce a 'merge commit' in the process. These merge commits rapidly build up and clutter up the history of the repository. For this reason, instead of pulling and introducing a merge commit you should do one of two things:
* Rebase your changes on top of the central repository. This can be achieved using 'git pull --rebase'.
* If your changes are significant, you can create a new named branch for them and merge that branch into the branch from the central repository.
In an ideal world, everyone would always follow these rules. However, in practice it is easy to forget to do so.

A solution is to block these merge commits from ever being pushed to the server. This is easily achieved using a git hook on the server side. The file 'update.merge' in the git-module repository gives an example of how to so. Place this file in the hooks directory of the repository on your server, rename it to 'update' and make sure it is executable. It will then block these annoying merge commits while allowing through genuine merge commits from named branches.

= Super repository =
With repository mirrors created and all modules now available as git repositories, it is quite convenient to have a container repository to manage everything. This container can be implemented as a git super-repository with git submodules for each component. In this section, we describe how to setup and administer such a repository, both locally and using a central server.

== Creating a repository ==

== Common tasks ==

=== Adding a submodule ===
Adding a new submodule is as simple as running:

git module add <url-to-repository> <path-to-submodule> [<upstream-url>] [<upstream-type>] [<track-branch>]

checking the changes look OK, committing and then pushing. The optional arguments <upstream-url>, <upstream-type> and <upstream-branch> specify the URL and type of the upstream version of this repository and the branch which the super-repository should track. If not given, <upstream-url> is assumed to be the same as <url-to-repository> and <upstream-type> is assumed to be 'git'. Other possible options for <upstream-type> are 'hg' and 'svn'. If <track-branch> is not given it defaults to 'trunk' for git-svn repositories and 'master' for git and hg-git repositories.

=== Removing a submodule ===
To remove a submodule from the super repository, use:

git module rm <submodule-path>

check the changes look OK, then commit and push.

''Include here a guide to setting up a Git server with a Cactus super-repository''

Have temporarily copy+pasted the README which is in note form - need to write a proper up-to-date guide.

<pre>

###############################################################################
# CactusGit
###############################################################################

# Introduction

# This package allows you to check out Cactus using GetComponents from
# a CRL thornlist using git-svn instead of svn for the svn
# repositories. This means that your local Cactus tree contains
# complete version information and history and can be managed using
# git rather than svn.

# Further, the package provides tools for creating a
# "super-repository" for the tree with pointers to the downloaded
# repositories as git submodules. This means that the state of the
# tree can be identified by a single commit ID and can be cloned
# directly without having to use GetComponents or git-svn.

# Notes
# * To make this more convenient, small wrapper scripts can be written
# to replace the recipes given below
#
# * GetComponents could be extended to support git-svn directly,
# rather than using the wrapper-script approach that is currently
# used

# Tutorial

# Add our tools to the path, including the svn wrapper which tricks
# GetComponents into making a git-svn clone of each svn repository.
# We also include a copy of GetComponents here for convenience.
export OLDPATH=$PATH PATH=$PWD/bin:$PATH

# Check out the tree into the Cactus directory (can take a very long
# time, ~60 minutes, and does not show progress for each repository
# until it has checked it out completely)
GetComponents -v -a WaveToy.th

# Now manually checkout any thorns which require authentication
git svn clone ...

# Add the git-svn repos to your server (also add the Cactus super-repo).
crlrepos WaveToy.th > gitsvn.list
cd <gitolite_dir>
vi conf/gitolite.conf # Add the repositories in gitsvn.list
git commit conf/gitolite.conf
git push

# Add a 'fetching and pushing' (fp) user
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

# Push as the fp user
scp Cactus fp@server:fetch/
scp gitsvn.list fp@server:fetch/
ssh fp@server
cd fetch
for i in `cat gitsvn.list`; do cd $i && git remote add bare cactus@localhost:$i && git config --unset remote.bare.fetch && git config remote.bare.push 'refs/remotes/*:refs/heads/*' && git push bare && cd -; done

# On the server, update the HEAD of the git-svn repos
ssh cactus@server
cd ~/repositories
for i in `cat ../fetch/gitsvn`; do cd $i.git && git symbolic-ref HEAD refs/heads/git-svn && cd -; done

# Generate a super-repository called CactusGit using all the
# repositories from Cactus listed in WaveToy.th
crltogit CactusGit cactus@server: Llama.CTGamma.ET.th

# Add non-svn repos
GetComponents Llama.CTGamma.ET.th
rm -rf repos
git submodule add ... repos/...

# Symlink flesh
ln -s flesh/CONTRIBUTORS
ln -s flesh/COPYRIGHT
ln -s flesh/Makefile
ln -s flesh/doc
ln -s flesh/lib

# Commit symlinks (as fp)
git add *
git commit ...
git push cactus@localhost:Cactus master:master

######### Administrators ########
# Updating the git-svn mirrors
for i in `cat gitsvn.list`; do cd $i; git svn fetch && git push bare; cd -; done

# Updating the hg-git mirrors
for i in `cat githg.list`; do cd $i; hg pull && hg push git; cd -; done

# Updating the git mirrors
for i in `cat git.list`; do cd $i; git fetch -a -u && git push bare --all; cd -; done

# Updating all submodules in the super-repo
cd Cactus
git pull
git submodule foreach git pull
git commit -m "Update submodules." -a

# Updating a single submodule in the super-repo
cd Cactus
cd <repo-path>
git pull
cd -
git commit -m "Update submodule <repo-path>." <repo-path>

# Automatic updates of super-repo
git clone --recursive cactus@localhost:Cactus
cd Cactus
git submodule foreach 'git remote set-url origin /var/cactus/repositories/$path cactus@git.barrywardell.net:/$path; echo OK'
git remote set-url origin /var/cactus/repositories/Cactus
git remote set-url --push origin cactus@git.barrywardell.net:Cactus
git checkout submodules
git pull origin submodules
git submodule foreach 'git fetch origin && git checkout origin'
git commit -m "Update submodules." -a
git push origin submodules

# Adding a git-svn repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi gitsvn.list # add <newrepo>
git svn clone -s <url> <newrepo>
cd <newrepo>
git remote add bare cactus@localhost:<newrepo>
git config --unset remote.bare.fetch
git config remote.bare.push 'refs/remotes/*:refs/heads/*'
git push bare
cd ~/repositories/<newrepo>
git symbolic-ref HEAD refs/heads/trunk
cd ~/fetch/<newrepo>
git pull bare

# Adding a hg-git repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi githg.list # add <newrepo>
hg clone <url> <newrepo>
cd <newrepo>
vi .hg/hgrc # Add:
# [path]
# git = git+ssh://localhost/<newrepo>
# [git]
# intree = 1
hg gexport # Can take quite a long time
hg push git

# Adding a git mirror repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi git.list # add <newrepo>
git clone --mirror <url> <newrepo>
cd <newrepo>
git remote add bare cactus@localhost:<newrepo>
git config --unset remote.bare.fetch
git push bare --all

# Adding a submodule to the super-repo
git submodule add <url> <path>
git commit
git push

# Removing a submodule from the super-repo
vi .gitmodules
vi .git/config
git rm --cached <path>
git commit ...
git push

# Changing svn URL for a git-svn clone. WARNING: this is rewriting the history of
# the git-svn branch of your mirror and will mess peoples checkout up.
cd fetch/<repo>
git status
git gc
git filter-branch --msg-filter 'sed "s/git-svn-id: http/git-svn-id: https/g"' $(cat .git/packed-refs | awk '// {print $2}' | grep -v 'pack-refs')
sed -i -e "s/http/https/g" .git/config
rm -rf .git/svn
git svn rebase
git push bare --force

# Changing svn URL without rewriting history (https://git.wiki.kernel.org/index.php/GitSvnSwitch):
Edit the svn-remote url URL in .git/config to point to the new domain name
Run git svn fetch - This needs to fetch at least one new revision from svn!
Change svn-remote url back to the original url
Run git svn rebase -l to do a local rebase (with the changes that came in with the last fetch operation)
Change svn-remote url back to the new url
Run git svn rebase should now work again!

</pre>

GitSuperRepoAdminGuide

2011-06-27T15:58:50Z

Barry.wardell: /* Local mirrors */

[[Category:GitSuperRepo]]

= Local mirrors =
The modules in a Cactus tree typically constitute a large number of repositories in various version control systems from wide number of sources. While GetComponents makes it get everything from their original locations, it may be still desirable to create local mirrors of these modules for several reasons:
* '''Speed''' - Some sources may be on the other side of the world so having a nearby local mirror can dramatically reduce the time to access them.
* '''Privacy''' - You may want to develop a new feature privately within your group before publicly releasing it. If this feature is based on an existing module then having a local mirror means you can create local private development branch which can later be easily merged and pushed upstream.
* '''Convenience''' - With remote repositories in a variety of version control systems, it is easy to get confused. Having local mirrors of these in a single version control system (i.e. git) simplifies things so that only a single tool is needed. In the case where the remote repository is in SVN local git mirrors also provides the advantage of offering the additional features git provides over svn.

This section describes how to setup, host and administer local mirrors of remote repositories.

== Creating repository mirrors ==

== Adding repository mirrors ==

== Removing repository mirrors ==

== Optional Extras ==
=== Blocking pushes of unnecessary merge commits ===

While working on a change in a git repository, it often happens that while you make local commits someone else pushes their own changes to the central repository. If you then try to push your changes, your push will be rejected as a non-fast-forward commit. The easiest solution is to pull the new changes from the server before pushing. However, this will introduce a 'merge commit' in the process. These merge commits rapidly build up and clutter up the history of the repository. For this reason, instead of pulling and introducing a merge commit you should do one of two things:
* Rebase your changes on top of the central repository. This can be achieved using 'git pull --rebase'.
* If your changes are significant, you can create a new named branch for them and merge that branch into the branch from the central repository.
In an ideal world, everyone would always follow these rules. However, in practice it is easy to forget to do so.

A solution is to block these merge commits from ever being pushed to the server. This is easily achieved using a git hook on the server side. The file 'update.merge' in the git-module repository gives an example of how to so. Place this file in the hooks directory of the repository on your server, rename it to 'update' and make sure it is executable. It will then block these annoying merge commits while allowing through genuine merge commits from named branches.

= Super repository =
With repository mirrors created and all modules now available as git repositories, it is quite convenient to have a container repository to manage everything. This container can be implemented as a git super-repository with git submodules for each component. In this section, we describe how to setup and administer such a repository, both locally and using a central server.

== Creating a repository ==

== Common tasks ==

=== Adding a submodule ===
Adding a new submodule is as simple as running:

git module add <url-to-repository> <path-to-submodule> [<upstream-url>] [<upstream-type>] [<track-branch>]

checking the changes look OK, committing and then pushing. The optional arguments <upstream-url>, <upstream-type> and <upstream-branch> specify the URL and type of the upstream version of this repository and the branch which the super-repository should track. If not given, <upstream-url> is assumed to be the same as <url-to-repository> and <upstream-type> is assumed to be 'git'. Other possible options for <upstream-type> are 'hg' and 'svn'. If <track-branch> is not given it defaults to 'trunk' for git-svn repositories and 'master' for git and hg-git repositories.

=== Removing a submodule ===
To remove a submodule from the super repository, use:

git module rm <submodule-path>

check the changes look OK, then commit and push.

''Include here a guide to setting up a Git server with a Cactus super-repository''

Have temporarily copy+pasted the README which is in note form - need to write a proper up-to-date guide.

<pre>

###############################################################################
# CactusGit
###############################################################################

# Introduction

# This package allows you to check out Cactus using GetComponents from
# a CRL thornlist using git-svn instead of svn for the svn
# repositories. This means that your local Cactus tree contains
# complete version information and history and can be managed using
# git rather than svn.

# Further, the package provides tools for creating a
# "super-repository" for the tree with pointers to the downloaded
# repositories as git submodules. This means that the state of the
# tree can be identified by a single commit ID and can be cloned
# directly without having to use GetComponents or git-svn.

# Notes
# * To make this more convenient, small wrapper scripts can be written
# to replace the recipes given below
#
# * GetComponents could be extended to support git-svn directly,
# rather than using the wrapper-script approach that is currently
# used

# Tutorial

# Add our tools to the path, including the svn wrapper which tricks
# GetComponents into making a git-svn clone of each svn repository.
# We also include a copy of GetComponents here for convenience.
export OLDPATH=$PATH PATH=$PWD/bin:$PATH

# Check out the tree into the Cactus directory (can take a very long
# time, ~60 minutes, and does not show progress for each repository
# until it has checked it out completely)
GetComponents -v -a WaveToy.th

# Now manually checkout any thorns which require authentication
git svn clone ...

# Add the git-svn repos to your server (also add the Cactus super-repo).
crlrepos WaveToy.th > gitsvn.list
cd <gitolite_dir>
vi conf/gitolite.conf # Add the repositories in gitsvn.list
git commit conf/gitolite.conf
git push

# Add a 'fetching and pushing' (fp) user
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

# Push as the fp user
scp Cactus fp@server:fetch/
scp gitsvn.list fp@server:fetch/
ssh fp@server
cd fetch
for i in `cat gitsvn.list`; do cd $i && git remote add bare cactus@localhost:$i && git config --unset remote.bare.fetch && git config remote.bare.push 'refs/remotes/*:refs/heads/*' && git push bare && cd -; done

# On the server, update the HEAD of the git-svn repos
ssh cactus@server
cd ~/repositories
for i in `cat ../fetch/gitsvn`; do cd $i.git && git symbolic-ref HEAD refs/heads/git-svn && cd -; done

# Generate a super-repository called CactusGit using all the
# repositories from Cactus listed in WaveToy.th
crltogit CactusGit cactus@server: Llama.CTGamma.ET.th

# Add non-svn repos
GetComponents Llama.CTGamma.ET.th
rm -rf repos
git submodule add ... repos/...

# Symlink flesh
ln -s flesh/CONTRIBUTORS
ln -s flesh/COPYRIGHT
ln -s flesh/Makefile
ln -s flesh/doc
ln -s flesh/lib

# Commit symlinks (as fp)
git add *
git commit ...
git push cactus@localhost:Cactus master:master

######### Administrators ########
# Updating the git-svn mirrors
for i in `cat gitsvn.list`; do cd $i; git svn fetch && git push bare; cd -; done

# Updating the hg-git mirrors
for i in `cat githg.list`; do cd $i; hg pull && hg push git; cd -; done

# Updating the git mirrors
for i in `cat git.list`; do cd $i; git fetch -a -u && git push bare --all; cd -; done

# Updating all submodules in the super-repo
cd Cactus
git pull
git submodule foreach git pull
git commit -m "Update submodules." -a

# Updating a single submodule in the super-repo
cd Cactus
cd <repo-path>
git pull
cd -
git commit -m "Update submodule <repo-path>." <repo-path>

# Automatic updates of super-repo
git clone --recursive cactus@localhost:Cactus
cd Cactus
git submodule foreach 'git remote set-url origin /var/cactus/repositories/$path cactus@git.barrywardell.net:/$path; echo OK'
git remote set-url origin /var/cactus/repositories/Cactus
git remote set-url --push origin cactus@git.barrywardell.net:Cactus
git checkout submodules
git pull origin submodules
git submodule foreach 'git fetch origin && git checkout origin'
git commit -m "Update submodules." -a
git push origin submodules

# Adding a git-svn repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi gitsvn.list # add <newrepo>
git svn clone -s <url> <newrepo>
cd <newrepo>
git remote add bare cactus@localhost:<newrepo>
git config --unset remote.bare.fetch
git config remote.bare.push 'refs/remotes/*:refs/heads/*'
git push bare
cd ~/repositories/<newrepo>
git symbolic-ref HEAD refs/heads/trunk
cd ~/fetch/<newrepo>
git pull bare

# Adding a hg-git repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi githg.list # add <newrepo>
hg clone <url> <newrepo>
cd <newrepo>
vi .hg/hgrc # Add:
# [path]
# git = git+ssh://localhost/<newrepo>
# [git]
# intree = 1
hg gexport # Can take quite a long time
hg push git

# Adding a git mirror repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi git.list # add <newrepo>
git clone --mirror <url> <newrepo>
cd <newrepo>
git remote add bare cactus@localhost:<newrepo>
git config --unset remote.bare.fetch
git push bare --all

# Adding a submodule to the super-repo
git submodule add <url> <path>
git commit
git push

# Removing a submodule from the super-repo
vi .gitmodules
vi .git/config
git rm --cached <path>
git commit ...
git push

# Changing svn URL for a git-svn clone. WARNING: this is rewriting the history of
# the git-svn branch of your mirror and will mess peoples checkout up.
cd fetch/<repo>
git status
git gc
git filter-branch --msg-filter 'sed "s/git-svn-id: http/git-svn-id: https/g"' $(cat .git/packed-refs | awk '// {print $2}' | grep -v 'pack-refs')
sed -i -e "s/http/https/g" .git/config
rm -rf .git/svn
git svn rebase
git push bare --force

# Changing svn URL without rewriting history (https://git.wiki.kernel.org/index.php/GitSvnSwitch):
Edit the svn-remote url URL in .git/config to point to the new domain name
Run git svn fetch - This needs to fetch at least one new revision from svn!
Change svn-remote url back to the original url
Run git svn rebase -l to do a local rebase (with the changes that came in with the last fetch operation)
Change svn-remote url back to the new url
Run git svn rebase should now work again!

</pre>

GitSuperRepoAdminGuide

2011-06-27T15:57:11Z

Barry.wardell: /* Removing a submodule */

[[Category:GitSuperRepo]]

= Local mirrors =
The modules in a Cactus tree typically constitute a large number of repositories in various version control systems from wide number of sources. It may be desirable to create local mirrors of these modules for several reasons:
* '''Speed''' - Some sources may be on the other side of the world so having a nearby local mirror can dramatically reduce the time to access them.
* '''Privacy''' - You may want to develop a new feature privately within your group before publicly releasing it. If this feature is based on an existing module then having a local mirror means you can create local private development branch which can later be easily merged and pushed upstream.
* '''Convenience''' - With remote repositories in a variety of version control systems, it is easy to get confused. Having local mirrors of these in a single version control system (i.e. git) simplifies things so that only a single tool is needed. In the case where the remote repository is in SVN local git mirrors also provides the advantage of offering the additional features git provides over svn.

This section describes how to setup, host and administer local mirrors of remote repositories.

== Creating repository mirrors ==

== Adding repository mirrors ==

== Removing repository mirrors ==

== Optional Extras ==
=== Blocking pushes of unnecessary merge commits ===

While working on a change in a git repository, it often happens that while you make local commits someone else pushes their own changes to the central repository. If you then try to push your changes, your push will be rejected as a non-fast-forward commit. The easiest solution is to pull the new changes from the server before pushing. However, this will introduce a 'merge commit' in the process. These merge commits rapidly build up and clutter up the history of the repository. For this reason, instead of pulling and introducing a merge commit you should do one of two things:
* Rebase your changes on top of the central repository. This can be achieved using 'git pull --rebase'.
* If your changes are significant, you can create a new named branch for them and merge that branch into the branch from the central repository.
In an ideal world, everyone would always follow these rules. However, in practice it is easy to forget to do so.

A solution is to block these merge commits from ever being pushed to the server. This is easily achieved using a git hook on the server side. The file 'update.merge' in the git-module repository gives an example of how to so. Place this file in the hooks directory of the repository on your server, rename it to 'update' and make sure it is executable. It will then block these annoying merge commits while allowing through genuine merge commits from named branches.

= Super repository =
With repository mirrors created and all modules now available as git repositories, it is quite convenient to have a container repository to manage everything. This container can be implemented as a git super-repository with git submodules for each component. In this section, we describe how to setup and administer such a repository, both locally and using a central server.

== Creating a repository ==

== Common tasks ==

=== Adding a submodule ===
Adding a new submodule is as simple as running:

git module add <url-to-repository> <path-to-submodule> [<upstream-url>] [<upstream-type>] [<track-branch>]

checking the changes look OK, committing and then pushing. The optional arguments <upstream-url>, <upstream-type> and <upstream-branch> specify the URL and type of the upstream version of this repository and the branch which the super-repository should track. If not given, <upstream-url> is assumed to be the same as <url-to-repository> and <upstream-type> is assumed to be 'git'. Other possible options for <upstream-type> are 'hg' and 'svn'. If <track-branch> is not given it defaults to 'trunk' for git-svn repositories and 'master' for git and hg-git repositories.

=== Removing a submodule ===
To remove a submodule from the super repository, use:

git module rm <submodule-path>

check the changes look OK, then commit and push.

''Include here a guide to setting up a Git server with a Cactus super-repository''

Have temporarily copy+pasted the README which is in note form - need to write a proper up-to-date guide.

<pre>

###############################################################################
# CactusGit
###############################################################################

# Introduction

# This package allows you to check out Cactus using GetComponents from
# a CRL thornlist using git-svn instead of svn for the svn
# repositories. This means that your local Cactus tree contains
# complete version information and history and can be managed using
# git rather than svn.

# Further, the package provides tools for creating a
# "super-repository" for the tree with pointers to the downloaded
# repositories as git submodules. This means that the state of the
# tree can be identified by a single commit ID and can be cloned
# directly without having to use GetComponents or git-svn.

# Notes
# * To make this more convenient, small wrapper scripts can be written
# to replace the recipes given below
#
# * GetComponents could be extended to support git-svn directly,
# rather than using the wrapper-script approach that is currently
# used

# Tutorial

# Add our tools to the path, including the svn wrapper which tricks
# GetComponents into making a git-svn clone of each svn repository.
# We also include a copy of GetComponents here for convenience.
export OLDPATH=$PATH PATH=$PWD/bin:$PATH

# Check out the tree into the Cactus directory (can take a very long
# time, ~60 minutes, and does not show progress for each repository
# until it has checked it out completely)
GetComponents -v -a WaveToy.th

# Now manually checkout any thorns which require authentication
git svn clone ...

# Add the git-svn repos to your server (also add the Cactus super-repo).
crlrepos WaveToy.th > gitsvn.list
cd <gitolite_dir>
vi conf/gitolite.conf # Add the repositories in gitsvn.list
git commit conf/gitolite.conf
git push

# Add a 'fetching and pushing' (fp) user
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

# Push as the fp user
scp Cactus fp@server:fetch/
scp gitsvn.list fp@server:fetch/
ssh fp@server
cd fetch
for i in `cat gitsvn.list`; do cd $i && git remote add bare cactus@localhost:$i && git config --unset remote.bare.fetch && git config remote.bare.push 'refs/remotes/*:refs/heads/*' && git push bare && cd -; done

# On the server, update the HEAD of the git-svn repos
ssh cactus@server
cd ~/repositories
for i in `cat ../fetch/gitsvn`; do cd $i.git && git symbolic-ref HEAD refs/heads/git-svn && cd -; done

# Generate a super-repository called CactusGit using all the
# repositories from Cactus listed in WaveToy.th
crltogit CactusGit cactus@server: Llama.CTGamma.ET.th

# Add non-svn repos
GetComponents Llama.CTGamma.ET.th
rm -rf repos
git submodule add ... repos/...

# Symlink flesh
ln -s flesh/CONTRIBUTORS
ln -s flesh/COPYRIGHT
ln -s flesh/Makefile
ln -s flesh/doc
ln -s flesh/lib

# Commit symlinks (as fp)
git add *
git commit ...
git push cactus@localhost:Cactus master:master

######### Administrators ########
# Updating the git-svn mirrors
for i in `cat gitsvn.list`; do cd $i; git svn fetch && git push bare; cd -; done

# Updating the hg-git mirrors
for i in `cat githg.list`; do cd $i; hg pull && hg push git; cd -; done

# Updating the git mirrors
for i in `cat git.list`; do cd $i; git fetch -a -u && git push bare --all; cd -; done

# Updating all submodules in the super-repo
cd Cactus
git pull
git submodule foreach git pull
git commit -m "Update submodules." -a

# Updating a single submodule in the super-repo
cd Cactus
cd <repo-path>
git pull
cd -
git commit -m "Update submodule <repo-path>." <repo-path>

# Automatic updates of super-repo
git clone --recursive cactus@localhost:Cactus
cd Cactus
git submodule foreach 'git remote set-url origin /var/cactus/repositories/$path cactus@git.barrywardell.net:/$path; echo OK'
git remote set-url origin /var/cactus/repositories/Cactus
git remote set-url --push origin cactus@git.barrywardell.net:Cactus
git checkout submodules
git pull origin submodules
git submodule foreach 'git fetch origin && git checkout origin'
git commit -m "Update submodules." -a
git push origin submodules

# Adding a git-svn repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi gitsvn.list # add <newrepo>
git svn clone -s <url> <newrepo>
cd <newrepo>
git remote add bare cactus@localhost:<newrepo>
git config --unset remote.bare.fetch
git config remote.bare.push 'refs/remotes/*:refs/heads/*'
git push bare
cd ~/repositories/<newrepo>
git symbolic-ref HEAD refs/heads/trunk
cd ~/fetch/<newrepo>
git pull bare

# Adding a hg-git repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi githg.list # add <newrepo>
hg clone <url> <newrepo>
cd <newrepo>
vi .hg/hgrc # Add:
# [path]
# git = git+ssh://localhost/<newrepo>
# [git]
# intree = 1
hg gexport # Can take quite a long time
hg push git

# Adding a git mirror repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi git.list # add <newrepo>
git clone --mirror <url> <newrepo>
cd <newrepo>
git remote add bare cactus@localhost:<newrepo>
git config --unset remote.bare.fetch
git push bare --all

# Adding a submodule to the super-repo
git submodule add <url> <path>
git commit
git push

# Removing a submodule from the super-repo
vi .gitmodules
vi .git/config
git rm --cached <path>
git commit ...
git push

# Changing svn URL for a git-svn clone. WARNING: this is rewriting the history of
# the git-svn branch of your mirror and will mess peoples checkout up.
cd fetch/<repo>
git status
git gc
git filter-branch --msg-filter 'sed "s/git-svn-id: http/git-svn-id: https/g"' $(cat .git/packed-refs | awk '// {print $2}' | grep -v 'pack-refs')
sed -i -e "s/http/https/g" .git/config
rm -rf .git/svn
git svn rebase
git push bare --force

# Changing svn URL without rewriting history (https://git.wiki.kernel.org/index.php/GitSvnSwitch):
Edit the svn-remote url URL in .git/config to point to the new domain name
Run git svn fetch - This needs to fetch at least one new revision from svn!
Change svn-remote url back to the original url
Run git svn rebase -l to do a local rebase (with the changes that came in with the last fetch operation)
Change svn-remote url back to the new url
Run git svn rebase should now work again!

</pre>

GitSuperRepoAdminGuide

2011-06-27T15:55:42Z

Barry.wardell: /* Local mirrors */

[[Category:GitSuperRepo]]

= Local mirrors =
The modules in a Cactus tree typically constitute a large number of repositories in various version control systems from wide number of sources. It may be desirable to create local mirrors of these modules for several reasons:
* '''Speed''' - Some sources may be on the other side of the world so having a nearby local mirror can dramatically reduce the time to access them.
* '''Privacy''' - You may want to develop a new feature privately within your group before publicly releasing it. If this feature is based on an existing module then having a local mirror means you can create local private development branch which can later be easily merged and pushed upstream.
* '''Convenience''' - With remote repositories in a variety of version control systems, it is easy to get confused. Having local mirrors of these in a single version control system (i.e. git) simplifies things so that only a single tool is needed. In the case where the remote repository is in SVN local git mirrors also provides the advantage of offering the additional features git provides over svn.

This section describes how to setup, host and administer local mirrors of remote repositories.

== Creating repository mirrors ==

== Adding repository mirrors ==

== Removing repository mirrors ==

== Optional Extras ==
=== Blocking pushes of unnecessary merge commits ===

While working on a change in a git repository, it often happens that while you make local commits someone else pushes their own changes to the central repository. If you then try to push your changes, your push will be rejected as a non-fast-forward commit. The easiest solution is to pull the new changes from the server before pushing. However, this will introduce a 'merge commit' in the process. These merge commits rapidly build up and clutter up the history of the repository. For this reason, instead of pulling and introducing a merge commit you should do one of two things:
* Rebase your changes on top of the central repository. This can be achieved using 'git pull --rebase'.
* If your changes are significant, you can create a new named branch for them and merge that branch into the branch from the central repository.
In an ideal world, everyone would always follow these rules. However, in practice it is easy to forget to do so.

A solution is to block these merge commits from ever being pushed to the server. This is easily achieved using a git hook on the server side. The file 'update.merge' in the git-module repository gives an example of how to so. Place this file in the hooks directory of the repository on your server, rename it to 'update' and make sure it is executable. It will then block these annoying merge commits while allowing through genuine merge commits from named branches.

= Super repository =
With repository mirrors created and all modules now available as git repositories, it is quite convenient to have a container repository to manage everything. This container can be implemented as a git super-repository with git submodules for each component. In this section, we describe how to setup and administer such a repository, both locally and using a central server.

== Creating a repository ==

== Common tasks ==

=== Adding a submodule ===
Adding a new submodule is as simple as running:

git module add <url-to-repository> <path-to-submodule> [<upstream-url>] [<upstream-type>] [<track-branch>]

checking the changes look OK, committing and then pushing. The optional arguments <upstream-url>, <upstream-type> and <upstream-branch> specify the URL and type of the upstream version of this repository and the branch which the super-repository should track. If not given, <upstream-url> is assumed to be the same as <url-to-repository> and <upstream-type> is assumed to be 'git'. Other possible options for <upstream-type> are 'hg' and 'svn'. If <track-branch> is not given it defaults to 'trunk' for git-svn repositories and 'master' for git and hg-git repositories.

=== Removing a submodule ===

''Include here a guide to setting up a Git server with a Cactus super-repository''

Have temporarily copy+pasted the README which is in note form - need to write a proper up-to-date guide.

<pre>

###############################################################################
# CactusGit
###############################################################################

# Introduction

# This package allows you to check out Cactus using GetComponents from
# a CRL thornlist using git-svn instead of svn for the svn
# repositories. This means that your local Cactus tree contains
# complete version information and history and can be managed using
# git rather than svn.

# Further, the package provides tools for creating a
# "super-repository" for the tree with pointers to the downloaded
# repositories as git submodules. This means that the state of the
# tree can be identified by a single commit ID and can be cloned
# directly without having to use GetComponents or git-svn.

# Notes
# * To make this more convenient, small wrapper scripts can be written
# to replace the recipes given below
#
# * GetComponents could be extended to support git-svn directly,
# rather than using the wrapper-script approach that is currently
# used

# Tutorial

# Add our tools to the path, including the svn wrapper which tricks
# GetComponents into making a git-svn clone of each svn repository.
# We also include a copy of GetComponents here for convenience.
export OLDPATH=$PATH PATH=$PWD/bin:$PATH

# Check out the tree into the Cactus directory (can take a very long
# time, ~60 minutes, and does not show progress for each repository
# until it has checked it out completely)
GetComponents -v -a WaveToy.th

# Now manually checkout any thorns which require authentication
git svn clone ...

# Add the git-svn repos to your server (also add the Cactus super-repo).
crlrepos WaveToy.th > gitsvn.list
cd <gitolite_dir>
vi conf/gitolite.conf # Add the repositories in gitsvn.list
git commit conf/gitolite.conf
git push

# Add a 'fetching and pushing' (fp) user
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

# Push as the fp user
scp Cactus fp@server:fetch/
scp gitsvn.list fp@server:fetch/
ssh fp@server
cd fetch
for i in `cat gitsvn.list`; do cd $i && git remote add bare cactus@localhost:$i && git config --unset remote.bare.fetch && git config remote.bare.push 'refs/remotes/*:refs/heads/*' && git push bare && cd -; done

# On the server, update the HEAD of the git-svn repos
ssh cactus@server
cd ~/repositories
for i in `cat ../fetch/gitsvn`; do cd $i.git && git symbolic-ref HEAD refs/heads/git-svn && cd -; done

# Generate a super-repository called CactusGit using all the
# repositories from Cactus listed in WaveToy.th
crltogit CactusGit cactus@server: Llama.CTGamma.ET.th

# Add non-svn repos
GetComponents Llama.CTGamma.ET.th
rm -rf repos
git submodule add ... repos/...

# Symlink flesh
ln -s flesh/CONTRIBUTORS
ln -s flesh/COPYRIGHT
ln -s flesh/Makefile
ln -s flesh/doc
ln -s flesh/lib

# Commit symlinks (as fp)
git add *
git commit ...
git push cactus@localhost:Cactus master:master

######### Administrators ########
# Updating the git-svn mirrors
for i in `cat gitsvn.list`; do cd $i; git svn fetch && git push bare; cd -; done

# Updating the hg-git mirrors
for i in `cat githg.list`; do cd $i; hg pull && hg push git; cd -; done

# Updating the git mirrors
for i in `cat git.list`; do cd $i; git fetch -a -u && git push bare --all; cd -; done

# Updating all submodules in the super-repo
cd Cactus
git pull
git submodule foreach git pull
git commit -m "Update submodules." -a

# Updating a single submodule in the super-repo
cd Cactus
cd <repo-path>
git pull
cd -
git commit -m "Update submodule <repo-path>." <repo-path>

# Automatic updates of super-repo
git clone --recursive cactus@localhost:Cactus
cd Cactus
git submodule foreach 'git remote set-url origin /var/cactus/repositories/$path cactus@git.barrywardell.net:/$path; echo OK'
git remote set-url origin /var/cactus/repositories/Cactus
git remote set-url --push origin cactus@git.barrywardell.net:Cactus
git checkout submodules
git pull origin submodules
git submodule foreach 'git fetch origin && git checkout origin'
git commit -m "Update submodules." -a
git push origin submodules

# Adding a git-svn repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi gitsvn.list # add <newrepo>
git svn clone -s <url> <newrepo>
cd <newrepo>
git remote add bare cactus@localhost:<newrepo>
git config --unset remote.bare.fetch
git config remote.bare.push 'refs/remotes/*:refs/heads/*'
git push bare
cd ~/repositories/<newrepo>
git symbolic-ref HEAD refs/heads/trunk
cd ~/fetch/<newrepo>
git pull bare

# Adding a hg-git repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi githg.list # add <newrepo>
hg clone <url> <newrepo>
cd <newrepo>
vi .hg/hgrc # Add:
# [path]
# git = git+ssh://localhost/<newrepo>
# [git]
# intree = 1
hg gexport # Can take quite a long time
hg push git

# Adding a git mirror repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi git.list # add <newrepo>
git clone --mirror <url> <newrepo>
cd <newrepo>
git remote add bare cactus@localhost:<newrepo>
git config --unset remote.bare.fetch
git push bare --all

# Adding a submodule to the super-repo
git submodule add <url> <path>
git commit
git push

# Removing a submodule from the super-repo
vi .gitmodules
vi .git/config
git rm --cached <path>
git commit ...
git push

# Changing svn URL for a git-svn clone. WARNING: this is rewriting the history of
# the git-svn branch of your mirror and will mess peoples checkout up.
cd fetch/<repo>
git status
git gc
git filter-branch --msg-filter 'sed "s/git-svn-id: http/git-svn-id: https/g"' $(cat .git/packed-refs | awk '// {print $2}' | grep -v 'pack-refs')
sed -i -e "s/http/https/g" .git/config
rm -rf .git/svn
git svn rebase
git push bare --force

# Changing svn URL without rewriting history (https://git.wiki.kernel.org/index.php/GitSvnSwitch):
Edit the svn-remote url URL in .git/config to point to the new domain name
Run git svn fetch - This needs to fetch at least one new revision from svn!
Change svn-remote url back to the original url
Run git svn rebase -l to do a local rebase (with the changes that came in with the last fetch operation)
Change svn-remote url back to the new url
Run git svn rebase should now work again!

</pre>

GitSuperRepo

2011-06-27T15:51:23Z

Barry.wardell: /* Documents */

[[Category:GitSuperRepo]]

The current method for checking out and updating components of the Einstein Toolkit has several shortcomings. We propose an additional layer of management around a Cactus tree achieved by storing it in a Git repository. This provides a uniform version control interface to the code with all the version control information from the source repositories available. It also allows the entire Cactus tree to be treated as a single repository, and allows "versions" of the code to be identified by a single Git revision.

This project is still in the early stages. We (Barry Wardell and Ian Hinder) are developing a set of techniques and tools which enable what we consider to be better workflows for using Cactus. As we use these ourselves, we will learn about what works and what doesn't, and hopefully will be able to provide recommended workflows for end-users.

==Documents==

* [[GitSuperRepoRationale|Rationale]]
* [[GitSuperRepoUsersGuide|User's guide]]
* [[GitSuperRepoAdminGuide|Server administrator's guide]]
* [[GitSuperRepoUnsorted|Unsorted information]]

==Progress==

* We have set up a repository which implements most of what is described in the [[GitSuperRepoRationale|Rationale]].

* Repository is updated from the source repositories every hour

==Planned work==

* Figure out some good scripts or git aliases for making working with the submodules more straightforward.

* Set up a public "einsteintoolkit" repository

* Integrate with build-and-test system so that there is always a branch which passes build-and-test

* Implement a technique for storing version control information (revision plus differences) into simulation output

* Update the group repository based on commit emails or an RSS feed rather than hourly

GitSuperRepoAdminGuide

2011-06-27T15:51:07Z

Barry.wardell: GitSuperRepoServerGuide moved to GitSuperRepoAdminGuide

[[Category:GitSuperRepo]]

= Local mirrors =
The modules in a Cactus tree typically constitute a large number of repositories in various version control systems from wide number of sources. It may be desirable to create local mirrors of these modules for several reasons:
* '''Speed''' - Some sources may be on the other side of the world so having a nearby local mirror can dramatically reduce the time to access them.
* '''Privacy''' - You may often want to develop a new feature privately within your group before releasing it. If this feature is based on an existing module then having a local mirror means you can create local private development which can later be easily merged and pushed upstream.
* '''Convenience''' - With remote repositories in a variety of version control systems, it is easy to get confused. Having local mirrors of these in a single version control system (i.e. git) simplifies things so that only a single tool is needed. In the case where the remote repository is in SVN local git mirrors also provides the advantage of offering the additional features git provides over svn.

This section describes how to setup, host and administer local mirrors of remote repositories.

== Creating repository mirrors ==

== Adding repository mirrors ==

== Removing repository mirrors ==

== Optional Extras ==
=== Blocking pushes of unnecessary merge commits ===

While working on a change in a git repository, it often happens that while you make local commits someone else pushes their own changes to the central repository. If you then try to push your changes, your push will be rejected as a non-fast-forward commit. The easiest solution is to pull the new changes from the server before pushing. However, this will introduce a 'merge commit' in the process. These merge commits rapidly build up and clutter up the history of the repository. For this reason, instead of pulling and introducing a merge commit you should do one of two things:
* Rebase your changes on top of the central repository. This can be achieved using 'git pull --rebase'.
* If your changes are significant, you can create a new named branch for them and merge that branch into the branch from the central repository.
In an ideal world, everyone would always follow these rules. However, in practice it is easy to forget to do so.

A solution is to block these merge commits from ever being pushed to the server. This is easily achieved using a git hook on the server side. The file 'update.merge' in the git-module repository gives an example of how to so. Place this file in the hooks directory of the repository on your server, rename it to 'update' and make sure it is executable. It will then block these annoying merge commits while allowing through genuine merge commits from named branches.

= Super repository =
With repository mirrors created and all modules now available as git repositories, it is quite convenient to have a container repository to manage everything. This container can be implemented as a git super-repository with git submodules for each component. In this section, we describe how to setup and administer such a repository, both locally and using a central server.

== Creating a repository ==

== Common tasks ==

=== Adding a submodule ===
Adding a new submodule is as simple as running:

git module add <url-to-repository> <path-to-submodule> [<upstream-url>] [<upstream-type>] [<track-branch>]

checking the changes look OK, committing and then pushing. The optional arguments <upstream-url>, <upstream-type> and <upstream-branch> specify the URL and type of the upstream version of this repository and the branch which the super-repository should track. If not given, <upstream-url> is assumed to be the same as <url-to-repository> and <upstream-type> is assumed to be 'git'. Other possible options for <upstream-type> are 'hg' and 'svn'. If <track-branch> is not given it defaults to 'trunk' for git-svn repositories and 'master' for git and hg-git repositories.

=== Removing a submodule ===

''Include here a guide to setting up a Git server with a Cactus super-repository''

Have temporarily copy+pasted the README which is in note form - need to write a proper up-to-date guide.

<pre>

###############################################################################
# CactusGit
###############################################################################

# Introduction

# This package allows you to check out Cactus using GetComponents from
# a CRL thornlist using git-svn instead of svn for the svn
# repositories. This means that your local Cactus tree contains
# complete version information and history and can be managed using
# git rather than svn.

# Further, the package provides tools for creating a
# "super-repository" for the tree with pointers to the downloaded
# repositories as git submodules. This means that the state of the
# tree can be identified by a single commit ID and can be cloned
# directly without having to use GetComponents or git-svn.

# Notes
# * To make this more convenient, small wrapper scripts can be written
# to replace the recipes given below
#
# * GetComponents could be extended to support git-svn directly,
# rather than using the wrapper-script approach that is currently
# used

# Tutorial

# Add our tools to the path, including the svn wrapper which tricks
# GetComponents into making a git-svn clone of each svn repository.
# We also include a copy of GetComponents here for convenience.
export OLDPATH=$PATH PATH=$PWD/bin:$PATH

# Check out the tree into the Cactus directory (can take a very long
# time, ~60 minutes, and does not show progress for each repository
# until it has checked it out completely)
GetComponents -v -a WaveToy.th

# Now manually checkout any thorns which require authentication
git svn clone ...

# Add the git-svn repos to your server (also add the Cactus super-repo).
crlrepos WaveToy.th > gitsvn.list
cd <gitolite_dir>
vi conf/gitolite.conf # Add the repositories in gitsvn.list
git commit conf/gitolite.conf
git push

# Add a 'fetching and pushing' (fp) user
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

# Push as the fp user
scp Cactus fp@server:fetch/
scp gitsvn.list fp@server:fetch/
ssh fp@server
cd fetch
for i in `cat gitsvn.list`; do cd $i && git remote add bare cactus@localhost:$i && git config --unset remote.bare.fetch && git config remote.bare.push 'refs/remotes/*:refs/heads/*' && git push bare && cd -; done

# On the server, update the HEAD of the git-svn repos
ssh cactus@server
cd ~/repositories
for i in `cat ../fetch/gitsvn`; do cd $i.git && git symbolic-ref HEAD refs/heads/git-svn && cd -; done

# Generate a super-repository called CactusGit using all the
# repositories from Cactus listed in WaveToy.th
crltogit CactusGit cactus@server: Llama.CTGamma.ET.th

# Add non-svn repos
GetComponents Llama.CTGamma.ET.th
rm -rf repos
git submodule add ... repos/...

# Symlink flesh
ln -s flesh/CONTRIBUTORS
ln -s flesh/COPYRIGHT
ln -s flesh/Makefile
ln -s flesh/doc
ln -s flesh/lib

# Commit symlinks (as fp)
git add *
git commit ...
git push cactus@localhost:Cactus master:master

######### Administrators ########
# Updating the git-svn mirrors
for i in `cat gitsvn.list`; do cd $i; git svn fetch && git push bare; cd -; done

# Updating the hg-git mirrors
for i in `cat githg.list`; do cd $i; hg pull && hg push git; cd -; done

# Updating the git mirrors
for i in `cat git.list`; do cd $i; git fetch -a -u && git push bare --all; cd -; done

# Updating all submodules in the super-repo
cd Cactus
git pull
git submodule foreach git pull
git commit -m "Update submodules." -a

# Updating a single submodule in the super-repo
cd Cactus
cd <repo-path>
git pull
cd -
git commit -m "Update submodule <repo-path>." <repo-path>

# Automatic updates of super-repo
git clone --recursive cactus@localhost:Cactus
cd Cactus
git submodule foreach 'git remote set-url origin /var/cactus/repositories/$path cactus@git.barrywardell.net:/$path; echo OK'
git remote set-url origin /var/cactus/repositories/Cactus
git remote set-url --push origin cactus@git.barrywardell.net:Cactus
git checkout submodules
git pull origin submodules
git submodule foreach 'git fetch origin && git checkout origin'
git commit -m "Update submodules." -a
git push origin submodules

# Adding a git-svn repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi gitsvn.list # add <newrepo>
git svn clone -s <url> <newrepo>
cd <newrepo>
git remote add bare cactus@localhost:<newrepo>
git config --unset remote.bare.fetch
git config remote.bare.push 'refs/remotes/*:refs/heads/*'
git push bare
cd ~/repositories/<newrepo>
git symbolic-ref HEAD refs/heads/trunk
cd ~/fetch/<newrepo>
git pull bare

# Adding a hg-git repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi githg.list # add <newrepo>
hg clone <url> <newrepo>
cd <newrepo>
vi .hg/hgrc # Add:
# [path]
# git = git+ssh://localhost/<newrepo>
# [git]
# intree = 1
hg gexport # Can take quite a long time
hg push git

# Adding a git mirror repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi git.list # add <newrepo>
git clone --mirror <url> <newrepo>
cd <newrepo>
git remote add bare cactus@localhost:<newrepo>
git config --unset remote.bare.fetch
git push bare --all

# Adding a submodule to the super-repo
git submodule add <url> <path>
git commit
git push

# Removing a submodule from the super-repo
vi .gitmodules
vi .git/config
git rm --cached <path>
git commit ...
git push

# Changing svn URL for a git-svn clone. WARNING: this is rewriting the history of
# the git-svn branch of your mirror and will mess peoples checkout up.
cd fetch/<repo>
git status
git gc
git filter-branch --msg-filter 'sed "s/git-svn-id: http/git-svn-id: https/g"' $(cat .git/packed-refs | awk '// {print $2}' | grep -v 'pack-refs')
sed -i -e "s/http/https/g" .git/config
rm -rf .git/svn
git svn rebase
git push bare --force

# Changing svn URL without rewriting history (https://git.wiki.kernel.org/index.php/GitSvnSwitch):
Edit the svn-remote url URL in .git/config to point to the new domain name
Run git svn fetch - This needs to fetch at least one new revision from svn!
Change svn-remote url back to the original url
Run git svn rebase -l to do a local rebase (with the changes that came in with the last fetch operation)
Change svn-remote url back to the new url
Run git svn rebase should now work again!

</pre>

GitSuperRepoServerGuide

2011-06-27T15:51:07Z

Barry.wardell: GitSuperRepoServerGuide moved to GitSuperRepoAdminGuide

#REDIRECT [[GitSuperRepoAdminGuide]]

GitSuperRepoAdminGuide

2011-06-27T15:50:05Z

Barry.wardell: /* Server Guide */

[[Category:GitSuperRepo]]

= Local mirrors =
The modules in a Cactus tree typically constitute a large number of repositories in various version control systems from wide number of sources. It may be desirable to create local mirrors of these modules for several reasons:
* '''Speed''' - Some sources may be on the other side of the world so having a nearby local mirror can dramatically reduce the time to access them.
* '''Privacy''' - You may often want to develop a new feature privately within your group before releasing it. If this feature is based on an existing module then having a local mirror means you can create local private development which can later be easily merged and pushed upstream.
* '''Convenience''' - With remote repositories in a variety of version control systems, it is easy to get confused. Having local mirrors of these in a single version control system (i.e. git) simplifies things so that only a single tool is needed. In the case where the remote repository is in SVN local git mirrors also provides the advantage of offering the additional features git provides over svn.

This section describes how to setup, host and administer local mirrors of remote repositories.

== Creating repository mirrors ==

== Adding repository mirrors ==

== Removing repository mirrors ==

== Optional Extras ==
=== Blocking pushes of unnecessary merge commits ===

While working on a change in a git repository, it often happens that while you make local commits someone else pushes their own changes to the central repository. If you then try to push your changes, your push will be rejected as a non-fast-forward commit. The easiest solution is to pull the new changes from the server before pushing. However, this will introduce a 'merge commit' in the process. These merge commits rapidly build up and clutter up the history of the repository. For this reason, instead of pulling and introducing a merge commit you should do one of two things:
* Rebase your changes on top of the central repository. This can be achieved using 'git pull --rebase'.
* If your changes are significant, you can create a new named branch for them and merge that branch into the branch from the central repository.
In an ideal world, everyone would always follow these rules. However, in practice it is easy to forget to do so.

A solution is to block these merge commits from ever being pushed to the server. This is easily achieved using a git hook on the server side. The file 'update.merge' in the git-module repository gives an example of how to so. Place this file in the hooks directory of the repository on your server, rename it to 'update' and make sure it is executable. It will then block these annoying merge commits while allowing through genuine merge commits from named branches.

= Super repository =
With repository mirrors created and all modules now available as git repositories, it is quite convenient to have a container repository to manage everything. This container can be implemented as a git super-repository with git submodules for each component. In this section, we describe how to setup and administer such a repository, both locally and using a central server.

== Creating a repository ==

== Common tasks ==

=== Adding a submodule ===
Adding a new submodule is as simple as running:

git module add <url-to-repository> <path-to-submodule> [<upstream-url>] [<upstream-type>] [<track-branch>]

checking the changes look OK, committing and then pushing. The optional arguments <upstream-url>, <upstream-type> and <upstream-branch> specify the URL and type of the upstream version of this repository and the branch which the super-repository should track. If not given, <upstream-url> is assumed to be the same as <url-to-repository> and <upstream-type> is assumed to be 'git'. Other possible options for <upstream-type> are 'hg' and 'svn'. If <track-branch> is not given it defaults to 'trunk' for git-svn repositories and 'master' for git and hg-git repositories.

=== Removing a submodule ===

''Include here a guide to setting up a Git server with a Cactus super-repository''

Have temporarily copy+pasted the README which is in note form - need to write a proper up-to-date guide.

<pre>

###############################################################################
# CactusGit
###############################################################################

# Introduction

# This package allows you to check out Cactus using GetComponents from
# a CRL thornlist using git-svn instead of svn for the svn
# repositories. This means that your local Cactus tree contains
# complete version information and history and can be managed using
# git rather than svn.

# Further, the package provides tools for creating a
# "super-repository" for the tree with pointers to the downloaded
# repositories as git submodules. This means that the state of the
# tree can be identified by a single commit ID and can be cloned
# directly without having to use GetComponents or git-svn.

# Notes
# * To make this more convenient, small wrapper scripts can be written
# to replace the recipes given below
#
# * GetComponents could be extended to support git-svn directly,
# rather than using the wrapper-script approach that is currently
# used

# Tutorial

# Add our tools to the path, including the svn wrapper which tricks
# GetComponents into making a git-svn clone of each svn repository.
# We also include a copy of GetComponents here for convenience.
export OLDPATH=$PATH PATH=$PWD/bin:$PATH

# Check out the tree into the Cactus directory (can take a very long
# time, ~60 minutes, and does not show progress for each repository
# until it has checked it out completely)
GetComponents -v -a WaveToy.th

# Now manually checkout any thorns which require authentication
git svn clone ...

# Add the git-svn repos to your server (also add the Cactus super-repo).
crlrepos WaveToy.th > gitsvn.list
cd <gitolite_dir>
vi conf/gitolite.conf # Add the repositories in gitsvn.list
git commit conf/gitolite.conf
git push

# Add a 'fetching and pushing' (fp) user
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

# Push as the fp user
scp Cactus fp@server:fetch/
scp gitsvn.list fp@server:fetch/
ssh fp@server
cd fetch
for i in `cat gitsvn.list`; do cd $i && git remote add bare cactus@localhost:$i && git config --unset remote.bare.fetch && git config remote.bare.push 'refs/remotes/*:refs/heads/*' && git push bare && cd -; done

# On the server, update the HEAD of the git-svn repos
ssh cactus@server
cd ~/repositories
for i in `cat ../fetch/gitsvn`; do cd $i.git && git symbolic-ref HEAD refs/heads/git-svn && cd -; done

# Generate a super-repository called CactusGit using all the
# repositories from Cactus listed in WaveToy.th
crltogit CactusGit cactus@server: Llama.CTGamma.ET.th

# Add non-svn repos
GetComponents Llama.CTGamma.ET.th
rm -rf repos
git submodule add ... repos/...

# Symlink flesh
ln -s flesh/CONTRIBUTORS
ln -s flesh/COPYRIGHT
ln -s flesh/Makefile
ln -s flesh/doc
ln -s flesh/lib

# Commit symlinks (as fp)
git add *
git commit ...
git push cactus@localhost:Cactus master:master

######### Administrators ########
# Updating the git-svn mirrors
for i in `cat gitsvn.list`; do cd $i; git svn fetch && git push bare; cd -; done

# Updating the hg-git mirrors
for i in `cat githg.list`; do cd $i; hg pull && hg push git; cd -; done

# Updating the git mirrors
for i in `cat git.list`; do cd $i; git fetch -a -u && git push bare --all; cd -; done

# Updating all submodules in the super-repo
cd Cactus
git pull
git submodule foreach git pull
git commit -m "Update submodules." -a

# Updating a single submodule in the super-repo
cd Cactus
cd <repo-path>
git pull
cd -
git commit -m "Update submodule <repo-path>." <repo-path>

# Automatic updates of super-repo
git clone --recursive cactus@localhost:Cactus
cd Cactus
git submodule foreach 'git remote set-url origin /var/cactus/repositories/$path cactus@git.barrywardell.net:/$path; echo OK'
git remote set-url origin /var/cactus/repositories/Cactus
git remote set-url --push origin cactus@git.barrywardell.net:Cactus
git checkout submodules
git pull origin submodules
git submodule foreach 'git fetch origin && git checkout origin'
git commit -m "Update submodules." -a
git push origin submodules

# Adding a git-svn repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi gitsvn.list # add <newrepo>
git svn clone -s <url> <newrepo>
cd <newrepo>
git remote add bare cactus@localhost:<newrepo>
git config --unset remote.bare.fetch
git config remote.bare.push 'refs/remotes/*:refs/heads/*'
git push bare
cd ~/repositories/<newrepo>
git symbolic-ref HEAD refs/heads/trunk
cd ~/fetch/<newrepo>
git pull bare

# Adding a hg-git repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi githg.list # add <newrepo>
hg clone <url> <newrepo>
cd <newrepo>
vi .hg/hgrc # Add:
# [path]
# git = git+ssh://localhost/<newrepo>
# [git]
# intree = 1
hg gexport # Can take quite a long time
hg push git

# Adding a git mirror repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi git.list # add <newrepo>
git clone --mirror <url> <newrepo>
cd <newrepo>
git remote add bare cactus@localhost:<newrepo>
git config --unset remote.bare.fetch
git push bare --all

# Adding a submodule to the super-repo
git submodule add <url> <path>
git commit
git push

# Removing a submodule from the super-repo
vi .gitmodules
vi .git/config
git rm --cached <path>
git commit ...
git push

# Changing svn URL for a git-svn clone. WARNING: this is rewriting the history of
# the git-svn branch of your mirror and will mess peoples checkout up.
cd fetch/<repo>
git status
git gc
git filter-branch --msg-filter 'sed "s/git-svn-id: http/git-svn-id: https/g"' $(cat .git/packed-refs | awk '// {print $2}' | grep -v 'pack-refs')
sed -i -e "s/http/https/g" .git/config
rm -rf .git/svn
git svn rebase
git push bare --force

# Changing svn URL without rewriting history (https://git.wiki.kernel.org/index.php/GitSvnSwitch):
Edit the svn-remote url URL in .git/config to point to the new domain name
Run git svn fetch - This needs to fetch at least one new revision from svn!
Change svn-remote url back to the original url
Run git svn rebase -l to do a local rebase (with the changes that came in with the last fetch operation)
Change svn-remote url back to the new url
Run git svn rebase should now work again!

</pre>

GitSuperRepoAdminGuide

2011-06-27T14:30:59Z

Barry.wardell: /* Optional Extras */

[[Category:GitSuperRepo]]

= Server Guide =

== Optional Extras ==
=== Blocking pushes of unnecessary merge commits ===

While working on a change in a git repository, it often happens that while you make local commits someone else pushes their own changes to the central repository. If you then try to push your changes, your push will be rejected as a non-fast-forward commit. The easiest solution is to pull the new changes from the server before pushing. However, this will introduce a 'merge commit' in the process. These merge commits rapidly build up and clutter up the history of the repository. For this reason, instead of pulling and introducing a merge commit you should do one of two things:
* Rebase your changes on top of the central repository. This can be achieved using 'git pull --rebase'.
* If your changes are significant, you can create a new named branch for them and merge that branch into the branch from the central repository.
In an ideal world, everyone would always follow these rules. However, in practice it is easy to forget to do so.

A solution is to block these merge commits from ever being pushed to the server. This is easily achieved using a git hook on the server side. The file 'update.merge' in the git-module repository gives an example of how to so. Place this file in the hooks directory of the repository on your server, rename it to 'update' and make sure it is executable. It will then block these annoying merge commits while allowing through genuine merge commits from named branches.

''Include here a guide to setting up a Git server with a Cactus super-repository''

Have temporarily copy+pasted the README which is in note form - need to write a proper up-to-date guide.

<pre>

###############################################################################
# CactusGit
###############################################################################

# Introduction

# This package allows you to check out Cactus using GetComponents from
# a CRL thornlist using git-svn instead of svn for the svn
# repositories. This means that your local Cactus tree contains
# complete version information and history and can be managed using
# git rather than svn.

# Further, the package provides tools for creating a
# "super-repository" for the tree with pointers to the downloaded
# repositories as git submodules. This means that the state of the
# tree can be identified by a single commit ID and can be cloned
# directly without having to use GetComponents or git-svn.

# Notes
# * To make this more convenient, small wrapper scripts can be written
# to replace the recipes given below
#
# * GetComponents could be extended to support git-svn directly,
# rather than using the wrapper-script approach that is currently
# used

# Tutorial

# Add our tools to the path, including the svn wrapper which tricks
# GetComponents into making a git-svn clone of each svn repository.
# We also include a copy of GetComponents here for convenience.
export OLDPATH=$PATH PATH=$PWD/bin:$PATH

# Check out the tree into the Cactus directory (can take a very long
# time, ~60 minutes, and does not show progress for each repository
# until it has checked it out completely)
GetComponents -v -a WaveToy.th

# Now manually checkout any thorns which require authentication
git svn clone ...

# Add the git-svn repos to your server (also add the Cactus super-repo).
crlrepos WaveToy.th > gitsvn.list
cd <gitolite_dir>
vi conf/gitolite.conf # Add the repositories in gitsvn.list
git commit conf/gitolite.conf
git push

# Add a 'fetching and pushing' (fp) user
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

# Push as the fp user
scp Cactus fp@server:fetch/
scp gitsvn.list fp@server:fetch/
ssh fp@server
cd fetch
for i in `cat gitsvn.list`; do cd $i && git remote add bare cactus@localhost:$i && git config --unset remote.bare.fetch && git config remote.bare.push 'refs/remotes/*:refs/heads/*' && git push bare && cd -; done

# On the server, update the HEAD of the git-svn repos
ssh cactus@server
cd ~/repositories
for i in `cat ../fetch/gitsvn`; do cd $i.git && git symbolic-ref HEAD refs/heads/git-svn && cd -; done

# Generate a super-repository called CactusGit using all the
# repositories from Cactus listed in WaveToy.th
crltogit CactusGit cactus@server: Llama.CTGamma.ET.th

# Add non-svn repos
GetComponents Llama.CTGamma.ET.th
rm -rf repos
git submodule add ... repos/...

# Symlink flesh
ln -s flesh/CONTRIBUTORS
ln -s flesh/COPYRIGHT
ln -s flesh/Makefile
ln -s flesh/doc
ln -s flesh/lib

# Commit symlinks (as fp)
git add *
git commit ...
git push cactus@localhost:Cactus master:master

######### Administrators ########
# Updating the git-svn mirrors
for i in `cat gitsvn.list`; do cd $i; git svn fetch && git push bare; cd -; done

# Updating the hg-git mirrors
for i in `cat githg.list`; do cd $i; hg pull && hg push git; cd -; done

# Updating the git mirrors
for i in `cat git.list`; do cd $i; git fetch -a -u && git push bare --all; cd -; done

# Updating all submodules in the super-repo
cd Cactus
git pull
git submodule foreach git pull
git commit -m "Update submodules." -a

# Updating a single submodule in the super-repo
cd Cactus
cd <repo-path>
git pull
cd -
git commit -m "Update submodule <repo-path>." <repo-path>

# Automatic updates of super-repo
git clone --recursive cactus@localhost:Cactus
cd Cactus
git submodule foreach 'git remote set-url origin /var/cactus/repositories/$path cactus@git.barrywardell.net:/$path; echo OK'
git remote set-url origin /var/cactus/repositories/Cactus
git remote set-url --push origin cactus@git.barrywardell.net:Cactus
git checkout submodules
git pull origin submodules
git submodule foreach 'git fetch origin && git checkout origin'
git commit -m "Update submodules." -a
git push origin submodules

# Adding a git-svn repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi gitsvn.list # add <newrepo>
git svn clone -s <url> <newrepo>
cd <newrepo>
git remote add bare cactus@localhost:<newrepo>
git config --unset remote.bare.fetch
git config remote.bare.push 'refs/remotes/*:refs/heads/*'
git push bare
cd ~/repositories/<newrepo>
git symbolic-ref HEAD refs/heads/trunk
cd ~/fetch/<newrepo>
git pull bare

# Adding a hg-git repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi githg.list # add <newrepo>
hg clone <url> <newrepo>
cd <newrepo>
vi .hg/hgrc # Add:
# [path]
# git = git+ssh://localhost/<newrepo>
# [git]
# intree = 1
hg gexport # Can take quite a long time
hg push git

# Adding a git mirror repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi git.list # add <newrepo>
git clone --mirror <url> <newrepo>
cd <newrepo>
git remote add bare cactus@localhost:<newrepo>
git config --unset remote.bare.fetch
git push bare --all

# Adding a submodule to the super-repo
git submodule add <url> <path>
git commit
git push

# Removing a submodule from the super-repo
vi .gitmodules
vi .git/config
git rm --cached <path>
git commit ...
git push

# Changing svn URL for a git-svn clone. WARNING: this is rewriting the history of
# the git-svn branch of your mirror and will mess peoples checkout up.
cd fetch/<repo>
git status
git gc
git filter-branch --msg-filter 'sed "s/git-svn-id: http/git-svn-id: https/g"' $(cat .git/packed-refs | awk '// {print $2}' | grep -v 'pack-refs')
sed -i -e "s/http/https/g" .git/config
rm -rf .git/svn
git svn rebase
git push bare --force

# Changing svn URL without rewriting history (https://git.wiki.kernel.org/index.php/GitSvnSwitch):
Edit the svn-remote url URL in .git/config to point to the new domain name
Run git svn fetch - This needs to fetch at least one new revision from svn!
Change svn-remote url back to the original url
Run git svn rebase -l to do a local rebase (with the changes that came in with the last fetch operation)
Change svn-remote url back to the new url
Run git svn rebase should now work again!

</pre>

GitSuperRepoAdminGuide

2011-06-27T14:30:12Z

Barry.wardell:

[[Category:GitSuperRepo]]

== Optional Extras ==
=== Blocking pushes of unnecessary merge commits ===

While working on a change in a git repository, it often happens that while you make local commits someone else pushes their own changes to the central repository. If you then try to push your changes, your push will be rejected as a non-fast-forward commit. The easiest solution is to pull the new changes from the server before pushing. However, this will introduce a 'merge commit' in the process. These merge commits rapidly build up and clutter up the history of the repository. For this reason, instead of pulling and introducing a merge commit you should do one of two things:
* Rebase your changes on top of the central repository. This can be achieved using 'git pull --rebase'.
* If your changes are significant, you can create a new named branch for them and merge that branch into the branch from the central repository.
In an ideal world, everyone would always follow these rules. However, in practice it is easy to forget to do so.

A solution is to block these merge commits from ever being pushed to the server. This is easily achieved using a git hook on the server side. The file 'update.merge' in the git-module repository gives an example of how to so. Place this file in the hooks directory of the repository on your server, rename it to 'update' and make sure it is executable. It will then block these annoying merge commits while allowing through genuine merge commits from named branches.

''Include here a guide to setting up a Git server with a Cactus super-repository''

Have temporarily copy+pasted the README which is in note form - need to write a proper up-to-date guide.

<pre>

###############################################################################
# CactusGit
###############################################################################

# Introduction

# This package allows you to check out Cactus using GetComponents from
# a CRL thornlist using git-svn instead of svn for the svn
# repositories. This means that your local Cactus tree contains
# complete version information and history and can be managed using
# git rather than svn.

# Further, the package provides tools for creating a
# "super-repository" for the tree with pointers to the downloaded
# repositories as git submodules. This means that the state of the
# tree can be identified by a single commit ID and can be cloned
# directly without having to use GetComponents or git-svn.

# Notes
# * To make this more convenient, small wrapper scripts can be written
# to replace the recipes given below
#
# * GetComponents could be extended to support git-svn directly,
# rather than using the wrapper-script approach that is currently
# used

# Tutorial

# Add our tools to the path, including the svn wrapper which tricks
# GetComponents into making a git-svn clone of each svn repository.
# We also include a copy of GetComponents here for convenience.
export OLDPATH=$PATH PATH=$PWD/bin:$PATH

# Check out the tree into the Cactus directory (can take a very long
# time, ~60 minutes, and does not show progress for each repository
# until it has checked it out completely)
GetComponents -v -a WaveToy.th

# Now manually checkout any thorns which require authentication
git svn clone ...

# Add the git-svn repos to your server (also add the Cactus super-repo).
crlrepos WaveToy.th > gitsvn.list
cd <gitolite_dir>
vi conf/gitolite.conf # Add the repositories in gitsvn.list
git commit conf/gitolite.conf
git push

# Add a 'fetching and pushing' (fp) user
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

# Push as the fp user
scp Cactus fp@server:fetch/
scp gitsvn.list fp@server:fetch/
ssh fp@server
cd fetch
for i in `cat gitsvn.list`; do cd $i && git remote add bare cactus@localhost:$i && git config --unset remote.bare.fetch && git config remote.bare.push 'refs/remotes/*:refs/heads/*' && git push bare && cd -; done

# On the server, update the HEAD of the git-svn repos
ssh cactus@server
cd ~/repositories
for i in `cat ../fetch/gitsvn`; do cd $i.git && git symbolic-ref HEAD refs/heads/git-svn && cd -; done

# Generate a super-repository called CactusGit using all the
# repositories from Cactus listed in WaveToy.th
crltogit CactusGit cactus@server: Llama.CTGamma.ET.th

# Add non-svn repos
GetComponents Llama.CTGamma.ET.th
rm -rf repos
git submodule add ... repos/...

# Symlink flesh
ln -s flesh/CONTRIBUTORS
ln -s flesh/COPYRIGHT
ln -s flesh/Makefile
ln -s flesh/doc
ln -s flesh/lib

# Commit symlinks (as fp)
git add *
git commit ...
git push cactus@localhost:Cactus master:master

######### Administrators ########
# Updating the git-svn mirrors
for i in `cat gitsvn.list`; do cd $i; git svn fetch && git push bare; cd -; done

# Updating the hg-git mirrors
for i in `cat githg.list`; do cd $i; hg pull && hg push git; cd -; done

# Updating the git mirrors
for i in `cat git.list`; do cd $i; git fetch -a -u && git push bare --all; cd -; done

# Updating all submodules in the super-repo
cd Cactus
git pull
git submodule foreach git pull
git commit -m "Update submodules." -a

# Updating a single submodule in the super-repo
cd Cactus
cd <repo-path>
git pull
cd -
git commit -m "Update submodule <repo-path>." <repo-path>

# Automatic updates of super-repo
git clone --recursive cactus@localhost:Cactus
cd Cactus
git submodule foreach 'git remote set-url origin /var/cactus/repositories/$path cactus@git.barrywardell.net:/$path; echo OK'
git remote set-url origin /var/cactus/repositories/Cactus
git remote set-url --push origin cactus@git.barrywardell.net:Cactus
git checkout submodules
git pull origin submodules
git submodule foreach 'git fetch origin && git checkout origin'
git commit -m "Update submodules." -a
git push origin submodules

# Adding a git-svn repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi gitsvn.list # add <newrepo>
git svn clone -s <url> <newrepo>
cd <newrepo>
git remote add bare cactus@localhost:<newrepo>
git config --unset remote.bare.fetch
git config remote.bare.push 'refs/remotes/*:refs/heads/*'
git push bare
cd ~/repositories/<newrepo>
git symbolic-ref HEAD refs/heads/trunk
cd ~/fetch/<newrepo>
git pull bare

# Adding a hg-git repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi githg.list # add <newrepo>
hg clone <url> <newrepo>
cd <newrepo>
vi .hg/hgrc # Add:
# [path]
# git = git+ssh://localhost/<newrepo>
# [git]
# intree = 1
hg gexport # Can take quite a long time
hg push git

# Adding a git mirror repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi git.list # add <newrepo>
git clone --mirror <url> <newrepo>
cd <newrepo>
git remote add bare cactus@localhost:<newrepo>
git config --unset remote.bare.fetch
git push bare --all

# Adding a submodule to the super-repo
git submodule add <url> <path>
git commit
git push

# Removing a submodule from the super-repo
vi .gitmodules
vi .git/config
git rm --cached <path>
git commit ...
git push

# Changing svn URL for a git-svn clone. WARNING: this is rewriting the history of
# the git-svn branch of your mirror and will mess peoples checkout up.
cd fetch/<repo>
git status
git gc
git filter-branch --msg-filter 'sed "s/git-svn-id: http/git-svn-id: https/g"' $(cat .git/packed-refs | awk '// {print $2}' | grep -v 'pack-refs')
sed -i -e "s/http/https/g" .git/config
rm -rf .git/svn
git svn rebase
git push bare --force

# Changing svn URL without rewriting history (https://git.wiki.kernel.org/index.php/GitSvnSwitch):
Edit the svn-remote url URL in .git/config to point to the new domain name
Run git svn fetch - This needs to fetch at least one new revision from svn!
Change svn-remote url back to the original url
Run git svn rebase -l to do a local rebase (with the changes that came in with the last fetch operation)
Change svn-remote url back to the new url
Run git svn rebase should now work again!

</pre>

GitSuperRepoUsersGuide

2011-06-27T14:12:22Z

Barry.wardell: /* Committing */

[[Category:GitSuperRepo]]

We provide a Git super-repo of the Einstein Toolkit. This page describes how to check it out and work with it. It is currently more of a tutorial than a User Guide.

==Checking out==

git clone --recursive cactus@git.einsteintoolkit.org:einsteintoolkit Cactus

This takes about 10 minutes and will create a directory called Cactus containing the entire Einstein Toolkit and all its version history. When it is done, setup the git-module package which provides convenience functionality:

cd Cactus
./bin/git-module setup && source ~/.profile

==Updating==

Before updating, it is first a good idea to see what would be changed. First enter the top-level super-repo directory and get any available updates:

git fetch && git submodule foreach git fetch

You can get a list of the changes using

git log --reverse --oneline -p --submodule master..origin/master $(git submodule|awk '{print $2}')

If this looks OK, you can then update using:

git pull
git submodule update

==Checking out a specific branch==

If you want to use a particular Einstein Toolkit release, perhaps because the current development version is unstable, use

git checkout ET_2011_06 && git module checkout --all

The tree will be very quickly updated to match the release. All changed files will have updated datestamps, so you should be able to trust the Cactus make system to recompile only what is necessary. However, it might be safer to delete any configurations and build them again.

==Committing==
For sub-repositories which are upstream (as most are), regardless of which revision control system is used upstream you must first initialise the sub-repository for committing:

git module init-upstream <submodule-path>

from the root Cactus (super-repo) directory. You can get a list of the available submodules with "git module ls" or by using tab completion if your shell is bash. Once you have completed this process for a submodule, you can treat the submodule as a regular git repository and commit as normal. The method of pushing depends on the version control system used upstream.

===Git===
For thorns whose source repositories are in Git already, you can commit and push as normal in the subrepositories.

git push upstream ...

===SVN===
For submodules which use SVN upstream, you can commit directly to the source SVN repository using, from the submodule directory,

git svn dcommit

assuming you have commit rights in the source SVN repository. This will push any local commits to SVN.

===Mercurial===

For Mercurial upstream repositories, it is more complicated, as there is no git-hg (equivalent to git-svn). In this case, you make commits locally to your submodule as if it were a regular git repository. Then, when you want to push upstream:

cd <submodule-path>.hg
hg pull git # Pull from the git submodule
hg push # Push to the upstream hg repo

==Viewing history==

In addition to the super repository being a git repository, each sub-repository is also a separate fully-fledged Git repository, so you can go into each one and type "git status", "git log", etc. This works for repositories which are SVN or Mercurial upstream, because we use git-svn and hg-git to convert the repositories to Git on the server. You can also use the usual graphical tools (we recommend [http://gitx.laullon.com/ GitX] for Mac OS, and git-gui/gitk for Linux) on each subrepository to visualise the log messages and patches, see which files have local changes, and interactively commit parts of files.

==Viewing local modifications==

You can see at a glance which of the submodules have local changes using

git status

You can then go into each submodule and see those changes using git status or git diff.

GitSuperRepoUsersGuide

2011-06-27T14:10:04Z

Barry.wardell: /* Viewing history */

[[Category:GitSuperRepo]]

We provide a Git super-repo of the Einstein Toolkit. This page describes how to check it out and work with it. It is currently more of a tutorial than a User Guide.

==Checking out==

git clone --recursive cactus@git.einsteintoolkit.org:einsteintoolkit Cactus

This takes about 10 minutes and will create a directory called Cactus containing the entire Einstein Toolkit and all its version history. When it is done, setup the git-module package which provides convenience functionality:

cd Cactus
./bin/git-module setup && source ~/.profile

==Updating==

Before updating, it is first a good idea to see what would be changed. First enter the top-level super-repo directory and get any available updates:

git fetch && git submodule foreach git fetch

You can get a list of the changes using

git log --reverse --oneline -p --submodule master..origin/master $(git submodule|awk '{print $2}')

If this looks OK, you can then update using:

git pull
git submodule update

==Checking out a specific branch==

If you want to use a particular Einstein Toolkit release, perhaps because the current development version is unstable, use

git checkout ET_2011_06 && git module checkout --all

The tree will be very quickly updated to match the release. All changed files will have updated datestamps, so you should be able to trust the Cactus make system to recompile only what is necessary. However, it might be safer to delete any configurations and build them again.

==Committing==
For sub-repositories which are upstream (as most are), regardless of which revision control system is used upstream you must first initialise the sub-repository for committing:

git module init-upstream <submodule-path>

from the root Cactus (super-repo) directory. You can get a list of the available submodules with "git module ls" or by using tab completion if your shell is bash. Once you have completed this process for a submodule, the method of pushing depends on the version control system used upstream.

===Git===
For thorns whose source repositories are in Git already, you can commit and push as normal in the subrepositories.

git push upstream ...

===SVN===
For submodules which use SVN upstream, you can commit directly to the source SVN repository using, from the submodule directory,

git svn dcommit

assuming you have commit rights in the source SVN repository. This will push any local commits to SVN.

===Mercurial===

For Mercurial upstream repositories, it is more complicated, as there is no git-hg (equivalent to git-svn). In this case, you make commits locally to your submodule as if it were a regular git repository. Then, when you want to push upstream:

cd <submodule-path>.hg
hg pull git # Pull from the git submodule
hg push # Push to the upstream hg repo

==Viewing history==

In addition to the super repository being a git repository, each sub-repository is also a separate fully-fledged Git repository, so you can go into each one and type "git status", "git log", etc. This works for repositories which are SVN or Mercurial upstream, because we use git-svn and hg-git to convert the repositories to Git on the server. You can also use the usual graphical tools (we recommend [http://gitx.laullon.com/ GitX] for Mac OS, and git-gui/gitk for Linux) on each subrepository to visualise the log messages and patches, see which files have local changes, and interactively commit parts of files.

==Viewing local modifications==

You can see at a glance which of the submodules have local changes using

git status

You can then go into each submodule and see those changes using git status or git diff.

GitSuperRepoUsersGuide

2011-06-27T14:09:40Z

Barry.wardell: /* Viewing history */

[[Category:GitSuperRepo]]

We provide a Git super-repo of the Einstein Toolkit. This page describes how to check it out and work with it. It is currently more of a tutorial than a User Guide.

==Checking out==

git clone --recursive cactus@git.einsteintoolkit.org:einsteintoolkit Cactus

This takes about 10 minutes and will create a directory called Cactus containing the entire Einstein Toolkit and all its version history. When it is done, setup the git-module package which provides convenience functionality:

cd Cactus
./bin/git-module setup && source ~/.profile

==Updating==

Before updating, it is first a good idea to see what would be changed. First enter the top-level super-repo directory and get any available updates:

git fetch && git submodule foreach git fetch

You can get a list of the changes using

git log --reverse --oneline -p --submodule master..origin/master $(git submodule|awk '{print $2}')

If this looks OK, you can then update using:

git pull
git submodule update

==Checking out a specific branch==

If you want to use a particular Einstein Toolkit release, perhaps because the current development version is unstable, use

git checkout ET_2011_06 && git module checkout --all

The tree will be very quickly updated to match the release. All changed files will have updated datestamps, so you should be able to trust the Cactus make system to recompile only what is necessary. However, it might be safer to delete any configurations and build them again.

==Committing==
For sub-repositories which are upstream (as most are), regardless of which revision control system is used upstream you must first initialise the sub-repository for committing:

git module init-upstream <submodule-path>

from the root Cactus (super-repo) directory. You can get a list of the available submodules with "git module ls" or by using tab completion if your shell is bash. Once you have completed this process for a submodule, the method of pushing depends on the version control system used upstream.

===Git===
For thorns whose source repositories are in Git already, you can commit and push as normal in the subrepositories.

git push upstream ...

===SVN===
For submodules which use SVN upstream, you can commit directly to the source SVN repository using, from the submodule directory,

git svn dcommit

assuming you have commit rights in the source SVN repository. This will push any local commits to SVN.

===Mercurial===

For Mercurial upstream repositories, it is more complicated, as there is no git-hg (equivalent to git-svn). In this case, you make commits locally to your submodule as if it were a regular git repository. Then, when you want to push upstream:

cd <submodule-path>.hg
hg pull git # Pull from the git submodule
hg push # Push to the upstream hg repo

==Viewing history==

In addition to the super repository being a git repository, each sub-repository is also a separate fully-fledged Git repository, so you can go into each one and type "git status", "git log", etc. This works for repositories which are SVN upstream, because we use git-svn to convert the repositories to Git on the server. You can also use the usual graphical tools (we recommend [http://gitx.laullon.com/ GitX] for Mac OS, and git-gui/gitk for Linux) on each subrepository to visualise the log messages and patches, see which files have local changes, and interactively commit parts of files.

==Viewing local modifications==

You can see at a glance which of the submodules have local changes using

git status

You can then go into each submodule and see those changes using git status or git diff.

GitSuperRepoUsersGuide

2011-06-27T14:09:27Z

Barry.wardell: /* Viewing history */

[[Category:GitSuperRepo]]

We provide a Git super-repo of the Einstein Toolkit. This page describes how to check it out and work with it. It is currently more of a tutorial than a User Guide.

==Checking out==

git clone --recursive cactus@git.einsteintoolkit.org:einsteintoolkit Cactus

This takes about 10 minutes and will create a directory called Cactus containing the entire Einstein Toolkit and all its version history. When it is done, setup the git-module package which provides convenience functionality:

cd Cactus
./bin/git-module setup && source ~/.profile

==Updating==

Before updating, it is first a good idea to see what would be changed. First enter the top-level super-repo directory and get any available updates:

git fetch && git submodule foreach git fetch

You can get a list of the changes using

git log --reverse --oneline -p --submodule master..origin/master $(git submodule|awk '{print $2}')

If this looks OK, you can then update using:

git pull
git submodule update

==Checking out a specific branch==

If you want to use a particular Einstein Toolkit release, perhaps because the current development version is unstable, use

git checkout ET_2011_06 && git module checkout --all

The tree will be very quickly updated to match the release. All changed files will have updated datestamps, so you should be able to trust the Cactus make system to recompile only what is necessary. However, it might be safer to delete any configurations and build them again.

==Committing==
For sub-repositories which are upstream (as most are), regardless of which revision control system is used upstream you must first initialise the sub-repository for committing:

git module init-upstream <submodule-path>

from the root Cactus (super-repo) directory. You can get a list of the available submodules with "git module ls" or by using tab completion if your shell is bash. Once you have completed this process for a submodule, the method of pushing depends on the version control system used upstream.

===Git===
For thorns whose source repositories are in Git already, you can commit and push as normal in the subrepositories.

git push upstream ...

===SVN===
For submodules which use SVN upstream, you can commit directly to the source SVN repository using, from the submodule directory,

git svn dcommit

assuming you have commit rights in the source SVN repository. This will push any local commits to SVN.

===Mercurial===

For Mercurial upstream repositories, it is more complicated, as there is no git-hg (equivalent to git-svn). In this case, you make commits locally to your submodule as if it were a regular git repository. Then, when you want to push upstream:

cd <submodule-path>.hg
hg pull git # Pull from the git submodule
hg push # Push to the upstream hg repo

==Viewing history==

In addition to the super repository being a git repository, each sub-repository is also a separate fully-fledged Git repository, so you can go into each one and type "git status", "git log" etc. This works for repositories which are SVN upstream, because we use git-svn to convert the repositories to Git on the server. You can also use the usual graphical tools (we recommend [http://gitx.laullon.com/ GitX] for Mac OS, and git-gui/gitk for Linux) on each subrepository to visualise the log messages and patches, see which files have local changes, and interactively commit parts of files.

==Viewing local modifications==

You can see at a glance which of the submodules have local changes using

git status

You can then go into each submodule and see those changes using git status or git diff.

GitSuperRepoUsersGuide

2011-06-27T14:09:11Z

Barry.wardell: /* Viewing history */

[[Category:GitSuperRepo]]

We provide a Git super-repo of the Einstein Toolkit. This page describes how to check it out and work with it. It is currently more of a tutorial than a User Guide.

==Checking out==

git clone --recursive cactus@git.einsteintoolkit.org:einsteintoolkit Cactus

This takes about 10 minutes and will create a directory called Cactus containing the entire Einstein Toolkit and all its version history. When it is done, setup the git-module package which provides convenience functionality:

cd Cactus
./bin/git-module setup && source ~/.profile

==Updating==

Before updating, it is first a good idea to see what would be changed. First enter the top-level super-repo directory and get any available updates:

git fetch && git submodule foreach git fetch

You can get a list of the changes using

git log --reverse --oneline -p --submodule master..origin/master $(git submodule|awk '{print $2}')

If this looks OK, you can then update using:

git pull
git submodule update

==Checking out a specific branch==

If you want to use a particular Einstein Toolkit release, perhaps because the current development version is unstable, use

git checkout ET_2011_06 && git module checkout --all

The tree will be very quickly updated to match the release. All changed files will have updated datestamps, so you should be able to trust the Cactus make system to recompile only what is necessary. However, it might be safer to delete any configurations and build them again.

==Committing==
For sub-repositories which are upstream (as most are), regardless of which revision control system is used upstream you must first initialise the sub-repository for committing:

git module init-upstream <submodule-path>

from the root Cactus (super-repo) directory. You can get a list of the available submodules with "git module ls" or by using tab completion if your shell is bash. Once you have completed this process for a submodule, the method of pushing depends on the version control system used upstream.

===Git===
For thorns whose source repositories are in Git already, you can commit and push as normal in the subrepositories.

git push upstream ...

===SVN===
For submodules which use SVN upstream, you can commit directly to the source SVN repository using, from the submodule directory,

git svn dcommit

assuming you have commit rights in the source SVN repository. This will push any local commits to SVN.

===Mercurial===

For Mercurial upstream repositories, it is more complicated, as there is no git-hg (equivalent to git-svn). In this case, you make commits locally to your submodule as if it were a regular git repository. Then, when you want to push upstream:

cd <submodule-path>.hg
hg pull git # Pull from the git submodule
hg push # Push to the upstream hg repo

==Viewing history==

In addition to the super repository being a git repository, each sub-repository is a separate fully-fledged Git repository, so you can go into each one and type "git status", "git log" etc. This works for repositories which are SVN upstream, because we use git-svn to convert the repositories to Git on the server. You can also use the usual graphical tools (we recommend [http://gitx.laullon.com/ GitX] for Mac OS, and git-gui/gitk for Linux) on each subrepository to visualise the log messages and patches, see which files have local changes, and interactively commit parts of files.

==Viewing local modifications==

You can see at a glance which of the submodules have local changes using

git status

You can then go into each submodule and see those changes using git status or git diff.

GitSuperRepoUsersGuide

2011-06-27T14:08:36Z

Barry.wardell: /* Viewing history */

[[Category:GitSuperRepo]]

We provide a Git super-repo of the Einstein Toolkit. This page describes how to check it out and work with it. It is currently more of a tutorial than a User Guide.

==Checking out==

git clone --recursive cactus@git.einsteintoolkit.org:einsteintoolkit Cactus

This takes about 10 minutes and will create a directory called Cactus containing the entire Einstein Toolkit and all its version history. When it is done, setup the git-module package which provides convenience functionality:

cd Cactus
./bin/git-module setup && source ~/.profile

==Updating==

Before updating, it is first a good idea to see what would be changed. First enter the top-level super-repo directory and get any available updates:

git fetch && git submodule foreach git fetch

You can get a list of the changes using

git log --reverse --oneline -p --submodule master..origin/master $(git submodule|awk '{print $2}')

If this looks OK, you can then update using:

git pull
git submodule update

==Checking out a specific branch==

If you want to use a particular Einstein Toolkit release, perhaps because the current development version is unstable, use

git checkout ET_2011_06 && git module checkout --all

The tree will be very quickly updated to match the release. All changed files will have updated datestamps, so you should be able to trust the Cactus make system to recompile only what is necessary. However, it might be safer to delete any configurations and build them again.

==Committing==
For sub-repositories which are upstream (as most are), regardless of which revision control system is used upstream you must first initialise the sub-repository for committing:

git module init-upstream <submodule-path>

from the root Cactus (super-repo) directory. You can get a list of the available submodules with "git module ls" or by using tab completion if your shell is bash. Once you have completed this process for a submodule, the method of pushing depends on the version control system used upstream.

===Git===
For thorns whose source repositories are in Git already, you can commit and push as normal in the subrepositories.

git push upstream ...

===SVN===
For submodules which use SVN upstream, you can commit directly to the source SVN repository using, from the submodule directory,

git svn dcommit

assuming you have commit rights in the source SVN repository. This will push any local commits to SVN.

===Mercurial===

For Mercurial upstream repositories, it is more complicated, as there is no git-hg (equivalent to git-svn). In this case, you make commits locally to your submodule as if it were a regular git repository. Then, when you want to push upstream:

cd <submodule-path>.hg
hg pull git # Pull from the git submodule
hg push # Push to the upstream hg repo

==Viewing history==

Each sub-repository is a separate fully-fledged Git repository, so you can go into each one and type "git status", "git log" etc. This works for repositories which are SVN upstream, because we use git-svn to convert the repositories to Git on the server. You can also use the usual graphical tools (we recommend [http://gitx.laullon.com/ GitX] for Mac OS, and git-gui/gitk for Linux) on each subrepository to visualise the log messages and patches, see which files have local changes, and interactively commit parts of files.

==Viewing local modifications==

You can see at a glance which of the submodules have local changes using

git status

You can then go into each submodule and see those changes using git status or git diff.

GitSuperRepoUsersGuide

2011-06-27T14:07:29Z

Barry.wardell: /* Viewing history */

[[Category:GitSuperRepo]]

We provide a Git super-repo of the Einstein Toolkit. This page describes how to check it out and work with it. It is currently more of a tutorial than a User Guide.

==Checking out==

git clone --recursive cactus@git.einsteintoolkit.org:einsteintoolkit Cactus

This takes about 10 minutes and will create a directory called Cactus containing the entire Einstein Toolkit and all its version history. When it is done, setup the git-module package which provides convenience functionality:

cd Cactus
./bin/git-module setup && source ~/.profile

==Updating==

Before updating, it is first a good idea to see what would be changed. First enter the top-level super-repo directory and get any available updates:

git fetch && git submodule foreach git fetch

You can get a list of the changes using

git log --reverse --oneline -p --submodule master..origin/master $(git submodule|awk '{print $2}')

If this looks OK, you can then update using:

git pull
git submodule update

==Checking out a specific branch==

If you want to use a particular Einstein Toolkit release, perhaps because the current development version is unstable, use

git checkout ET_2011_06 && git module checkout --all

The tree will be very quickly updated to match the release. All changed files will have updated datestamps, so you should be able to trust the Cactus make system to recompile only what is necessary. However, it might be safer to delete any configurations and build them again.

==Committing==
For sub-repositories which are upstream (as most are), regardless of which revision control system is used upstream you must first initialise the sub-repository for committing:

git module init-upstream <submodule-path>

from the root Cactus (super-repo) directory. You can get a list of the available submodules with "git module ls" or by using tab completion if your shell is bash. Once you have completed this process for a submodule, the method of pushing depends on the version control system used upstream.

===Git===
For thorns whose source repositories are in Git already, you can commit and push as normal in the subrepositories.

git push upstream ...

===SVN===
For submodules which use SVN upstream, you can commit directly to the source SVN repository using, from the submodule directory,

git svn dcommit

assuming you have commit rights in the source SVN repository. This will push any local commits to SVN.

===Mercurial===

For Mercurial upstream repositories, it is more complicated, as there is no git-hg (equivalent to git-svn). In this case, you make commits locally to your submodule as if it were a regular git repository. Then, when you want to push upstream:

cd <submodule-path>.hg
hg pull git # Pull from the git submodule
hg push # Push to the upstream hg repo

==Viewing history==

Each sub-repository is a separate fully-fledged Git repository, so you can go into each one and type "git status", "git log" etc. This works for repositories which are SVN upstream, because we use git-svn to convert the repositories to Git on the server. You can also use the usual graphical tools (we recommend [GitX|http://gitx.laullon.com/] for Mac OS, and git-gui/gitk for Linux) on each subrepository to visualise the log messages and patches, see which files have local changes, and interactively commit parts of files.

==Viewing local modifications==

You can see at a glance which of the submodules have local changes using

git status

You can then go into each submodule and see those changes using git status or git diff.

GitSuperRepoUsersGuide

2011-06-27T14:06:44Z

Barry.wardell: /* Viewing history */

[[Category:GitSuperRepo]]

We provide a Git super-repo of the Einstein Toolkit. This page describes how to check it out and work with it. It is currently more of a tutorial than a User Guide.

==Checking out==

git clone --recursive cactus@git.einsteintoolkit.org:einsteintoolkit Cactus

This takes about 10 minutes and will create a directory called Cactus containing the entire Einstein Toolkit and all its version history. When it is done, setup the git-module package which provides convenience functionality:

cd Cactus
./bin/git-module setup && source ~/.profile

==Updating==

Before updating, it is first a good idea to see what would be changed. First enter the top-level super-repo directory and get any available updates:

git fetch && git submodule foreach git fetch

You can get a list of the changes using

git log --reverse --oneline -p --submodule master..origin/master $(git submodule|awk '{print $2}')

If this looks OK, you can then update using:

git pull
git submodule update

==Checking out a specific branch==

If you want to use a particular Einstein Toolkit release, perhaps because the current development version is unstable, use

git checkout ET_2011_06 && git module checkout --all

The tree will be very quickly updated to match the release. All changed files will have updated datestamps, so you should be able to trust the Cactus make system to recompile only what is necessary. However, it might be safer to delete any configurations and build them again.

==Committing==
For sub-repositories which are upstream (as most are), regardless of which revision control system is used upstream you must first initialise the sub-repository for committing:

git module init-upstream <submodule-path>

from the root Cactus (super-repo) directory. You can get a list of the available submodules with "git module ls" or by using tab completion if your shell is bash. Once you have completed this process for a submodule, the method of pushing depends on the version control system used upstream.

===Git===
For thorns whose source repositories are in Git already, you can commit and push as normal in the subrepositories.

git push upstream ...

===SVN===
For submodules which use SVN upstream, you can commit directly to the source SVN repository using, from the submodule directory,

git svn dcommit

assuming you have commit rights in the source SVN repository. This will push any local commits to SVN.

===Mercurial===

For Mercurial upstream repositories, it is more complicated, as there is no git-hg (equivalent to git-svn). In this case, you make commits locally to your submodule as if it were a regular git repository. Then, when you want to push upstream:

cd <submodule-path>.hg
hg pull git # Pull from the git submodule
hg push # Push to the upstream hg repo

==Viewing history==

Each sub-repository is a separate fully-fledged Git repository, so you can go into each one and type "git status", "git log" etc. This works for repositories which are SVN upstream, because we use git-svn to convert the repositories to Git on the server. You can also use the usual graphical tools (we recommend [http://gitx.laullon.com/|GitX] for Mac OS, and git-gui/gitk for Linux) on each subrepository to visualise the log messages and patches, see which files have local changes, and interactively commit parts of files.

==Viewing local modifications==

You can see at a glance which of the submodules have local changes using

git status

You can then go into each submodule and see those changes using git status or git diff.

GitSuperRepoUsersGuide

2011-06-27T14:03:53Z

Barry.wardell: /* Updating */

[[Category:GitSuperRepo]]

We provide a Git super-repo of the Einstein Toolkit. This page describes how to check it out and work with it. It is currently more of a tutorial than a User Guide.

==Checking out==

git clone --recursive cactus@git.einsteintoolkit.org:einsteintoolkit Cactus

This takes about 10 minutes and will create a directory called Cactus containing the entire Einstein Toolkit and all its version history. When it is done, setup the git-module package which provides convenience functionality:

cd Cactus
./bin/git-module setup && source ~/.profile

==Updating==

Before updating, it is first a good idea to see what would be changed. First enter the top-level super-repo directory and get any available updates:

git fetch && git submodule foreach git fetch

You can get a list of the changes using

git log --reverse --oneline -p --submodule master..origin/master $(git submodule|awk '{print $2}')

If this looks OK, you can then update using:

git pull
git submodule update

==Checking out a specific branch==

If you want to use a particular Einstein Toolkit release, perhaps because the current development version is unstable, use

git checkout ET_2011_06 && git module checkout --all

The tree will be very quickly updated to match the release. All changed files will have updated datestamps, so you should be able to trust the Cactus make system to recompile only what is necessary. However, it might be safer to delete any configurations and build them again.

==Committing==
For sub-repositories which are upstream (as most are), regardless of which revision control system is used upstream you must first initialise the sub-repository for committing:

git module init-upstream <submodule-path>

from the root Cactus (super-repo) directory. You can get a list of the available submodules with "git module ls" or by using tab completion if your shell is bash. Once you have completed this process for a submodule, the method of pushing depends on the version control system used upstream.

===Git===
For thorns whose source repositories are in Git already, you can commit and push as normal in the subrepositories.

git push upstream ...

===SVN===
For submodules which use SVN upstream, you can commit directly to the source SVN repository using, from the submodule directory,

git svn dcommit

assuming you have commit rights in the source SVN repository. This will push any local commits to SVN.

===Mercurial===

For Mercurial upstream repositories, it is more complicated, as there is no git-hg (equivalent to git-svn). In this case, you make commits locally to your submodule as if it were a regular git repository. Then, when you want to push upstream:

cd <submodule-path>.hg
hg pull git # Pull from the git submodule
hg push # Push to the upstream hg repo

==Viewing history==

Each sub-repository is a separate fully-fledged Git repository, so you can go into each one and type "git status", "git log" etc. This works for repositories which are SVN upstream, because we use git-svn to convert the repositories to Git on the server. You can also use the usual graphical tools (we recommend GitX from [http://brotherbard.com/blog/2010/03/experimental-gitx-fork/] for Mac OS, and XXX for Linux) on each subrepository to visualise the log messages and patches, see which files have local changes, and interactively commit parts of files.

==Viewing local modifications==

You can see at a glance which of the submodules have local changes using

git status

You can then go into each submodule and see those changes using git status or git diff.

GitSuperRepoUsersGuide

2011-06-27T13:58:03Z

Barry.wardell: /* Mercurial */

[[Category:GitSuperRepo]]

We provide a Git super-repo of the Einstein Toolkit. This page describes how to check it out and work with it. It is currently more of a tutorial than a User Guide.

==Checking out==

git clone --recursive cactus@git.einsteintoolkit.org:einsteintoolkit Cactus

This takes about 10 minutes and will create a directory called Cactus containing the entire Einstein Toolkit and all its version history. When it is done, setup the git-module package which provides convenience functionality:

cd Cactus
./bin/git-module setup && source ~/.profile

==Updating==

To update the checkout, it is first a good idea to see what would be changed. First enter the top-level super-repo directory. You can get a list of the changes using

git submodule foreach 'git --no-pager log --oneline ..remotes/origin/HEAD'

This assumes that the remote HEAD is the one that is checked out, which is not true for SimFactory, for example. Also, don't we need to do some fetching first?

If this is OK, you can then update using:

git pull
git submodule update

==Checking out a specific branch==

If you want to use a particular Einstein Toolkit release, perhaps because the current development version is unstable, use

git checkout ET_2011_06 && git module checkout --all

The tree will be very quickly updated to match the release. All changed files will have updated datestamps, so you should be able to trust the Cactus make system to recompile only what is necessary. However, it might be safer to delete any configurations and build them again.

==Committing==
For sub-repositories which are upstream (as most are), regardless of which revision control system is used upstream you must first initialise the sub-repository for committing:

git module init-upstream <submodule-path>

from the root Cactus (super-repo) directory. You can get a list of the available submodules with "git module ls" or by using tab completion if your shell is bash. Once you have completed this process for a submodule, the method of pushing depends on the version control system used upstream.

===Git===
For thorns whose source repositories are in Git already, you can commit and push as normal in the subrepositories.

git push upstream ...

===SVN===
For submodules which use SVN upstream, you can commit directly to the source SVN repository using, from the submodule directory,

git svn dcommit

assuming you have commit rights in the source SVN repository. This will push any local commits to SVN.

===Mercurial===

For Mercurial upstream repositories, it is more complicated, as there is no git-hg (equivalent to git-svn). In this case, you make commits locally to your submodule as if it were a regular git repository. Then, when you want to push upstream:

cd <submodule-path>.hg
hg pull git # Pull from the git submodule
hg push # Push to the upstream hg repo

==Viewing history==

Each sub-repository is a separate fully-fledged Git repository, so you can go into each one and type "git status", "git log" etc. This works for repositories which are SVN upstream, because we use git-svn to convert the repositories to Git on the server. You can also use the usual graphical tools (we recommend GitX from [http://brotherbard.com/blog/2010/03/experimental-gitx-fork/] for Mac OS, and XXX for Linux) on each subrepository to visualise the log messages and patches, see which files have local changes, and interactively commit parts of files.

==Viewing local modifications==

You can see at a glance which of the submodules have local changes using

git status

You can then go into each submodule and see those changes using git status or git diff.

GitSuperRepoUsersGuide

2011-06-27T13:57:23Z

Barry.wardell: /* Checking out a specific branch */

[[Category:GitSuperRepo]]

We provide a Git super-repo of the Einstein Toolkit. This page describes how to check it out and work with it. It is currently more of a tutorial than a User Guide.

==Checking out==

git clone --recursive cactus@git.einsteintoolkit.org:einsteintoolkit Cactus

This takes about 10 minutes and will create a directory called Cactus containing the entire Einstein Toolkit and all its version history. When it is done, setup the git-module package which provides convenience functionality:

cd Cactus
./bin/git-module setup && source ~/.profile

==Updating==

To update the checkout, it is first a good idea to see what would be changed. First enter the top-level super-repo directory. You can get a list of the changes using

git submodule foreach 'git --no-pager log --oneline ..remotes/origin/HEAD'

This assumes that the remote HEAD is the one that is checked out, which is not true for SimFactory, for example. Also, don't we need to do some fetching first?

If this is OK, you can then update using:

git pull
git submodule update

==Checking out a specific branch==

If you want to use a particular Einstein Toolkit release, perhaps because the current development version is unstable, use

git checkout ET_2011_06 && git module checkout --all

The tree will be very quickly updated to match the release. All changed files will have updated datestamps, so you should be able to trust the Cactus make system to recompile only what is necessary. However, it might be safer to delete any configurations and build them again.

==Committing==
For sub-repositories which are upstream (as most are), regardless of which revision control system is used upstream you must first initialise the sub-repository for committing:

git module init-upstream <submodule-path>

from the root Cactus (super-repo) directory. You can get a list of the available submodules with "git module ls" or by using tab completion if your shell is bash. Once you have completed this process for a submodule, the method of pushing depends on the version control system used upstream.

===Git===
For thorns whose source repositories are in Git already, you can commit and push as normal in the subrepositories.

git push upstream ...

===SVN===
For submodules which use SVN upstream, you can commit directly to the source SVN repository using, from the submodule directory,

git svn dcommit

assuming you have commit rights in the source SVN repository. This will push any local commits to SVN.

===Mercurial===

For Mercurial upstream repositories, it is more complicated, as there is no git-hg (equivalent to git-svn). In this case, you make commits locally to your submodule as if it were a regular git repository. Then, when you want to push upstream:

cd <submodule-path>.hg
hg pull git # Pull from the git submodule
hg push # Push to the upstream hg repo

This obviously has to be improved.

==Viewing history==

Each sub-repository is a separate fully-fledged Git repository, so you can go into each one and type "git status", "git log" etc. This works for repositories which are SVN upstream, because we use git-svn to convert the repositories to Git on the server. You can also use the usual graphical tools (we recommend GitX from [http://brotherbard.com/blog/2010/03/experimental-gitx-fork/] for Mac OS, and XXX for Linux) on each subrepository to visualise the log messages and patches, see which files have local changes, and interactively commit parts of files.

==Viewing local modifications==

You can see at a glance which of the submodules have local changes using

git status

You can then go into each submodule and see those changes using git status or git diff.

GitSuperRepoUsersGuide

2011-06-27T13:57:05Z

Barry.wardell: /* Checking out a specific branch */

[[Category:GitSuperRepo]]

We provide a Git super-repo of the Einstein Toolkit. This page describes how to check it out and work with it. It is currently more of a tutorial than a User Guide.

==Checking out==

git clone --recursive cactus@git.einsteintoolkit.org:einsteintoolkit Cactus

This takes about 10 minutes and will create a directory called Cactus containing the entire Einstein Toolkit and all its version history. When it is done, setup the git-module package which provides convenience functionality:

cd Cactus
./bin/git-module setup && source ~/.profile

==Updating==

To update the checkout, it is first a good idea to see what would be changed. First enter the top-level super-repo directory. You can get a list of the changes using

git submodule foreach 'git --no-pager log --oneline ..remotes/origin/HEAD'

This assumes that the remote HEAD is the one that is checked out, which is not true for SimFactory, for example. Also, don't we need to do some fetching first?

If this is OK, you can then update using:

git pull
git submodule update

==Checking out a specific branch==

If you want to use a particular Einstein Toolkit release, perhaps because the current development version is unstable, use

git checkout ET_2011_06 && git module checkout

The tree will be very quickly updated to match the release. All changed files will have updated datestamps, so you should be able to trust the Cactus make system to recompile only what is necessary. However, it might be safer to delete any configurations and build them again.

==Committing==
For sub-repositories which are upstream (as most are), regardless of which revision control system is used upstream you must first initialise the sub-repository for committing:

git module init-upstream <submodule-path>

from the root Cactus (super-repo) directory. You can get a list of the available submodules with "git module ls" or by using tab completion if your shell is bash. Once you have completed this process for a submodule, the method of pushing depends on the version control system used upstream.

===Git===
For thorns whose source repositories are in Git already, you can commit and push as normal in the subrepositories.

git push upstream ...

===SVN===
For submodules which use SVN upstream, you can commit directly to the source SVN repository using, from the submodule directory,

git svn dcommit

assuming you have commit rights in the source SVN repository. This will push any local commits to SVN.

===Mercurial===

For Mercurial upstream repositories, it is more complicated, as there is no git-hg (equivalent to git-svn). In this case, you make commits locally to your submodule as if it were a regular git repository. Then, when you want to push upstream:

cd <submodule-path>.hg
hg pull git # Pull from the git submodule
hg push # Push to the upstream hg repo

This obviously has to be improved.

==Viewing history==

Each sub-repository is a separate fully-fledged Git repository, so you can go into each one and type "git status", "git log" etc. This works for repositories which are SVN upstream, because we use git-svn to convert the repositories to Git on the server. You can also use the usual graphical tools (we recommend GitX from [http://brotherbard.com/blog/2010/03/experimental-gitx-fork/] for Mac OS, and XXX for Linux) on each subrepository to visualise the log messages and patches, see which files have local changes, and interactively commit parts of files.

==Viewing local modifications==

You can see at a glance which of the submodules have local changes using

git status

You can then go into each submodule and see those changes using git status or git diff.

GitSuperRepoUsersGuide

2011-06-27T13:55:11Z

Barry.wardell:

[[Category:GitSuperRepo]]

We provide a Git super-repo of the Einstein Toolkit. This page describes how to check it out and work with it. It is currently more of a tutorial than a User Guide.

==Checking out==

git clone --recursive cactus@git.einsteintoolkit.org:einsteintoolkit Cactus

This takes about 10 minutes and will create a directory called Cactus containing the entire Einstein Toolkit and all its version history. When it is done, setup the git-module package which provides convenience functionality:

cd Cactus
./bin/git-module setup && source ~/.profile

==Updating==

To update the checkout, it is first a good idea to see what would be changed. First enter the top-level super-repo directory. You can get a list of the changes using

git submodule foreach 'git --no-pager log --oneline ..remotes/origin/HEAD'

This assumes that the remote HEAD is the one that is checked out, which is not true for SimFactory, for example. Also, don't we need to do some fetching first?

If this is OK, you can then update using:

git pull
git submodule update

==Checking out a specific branch==

If you want to use a particular Einstein Toolkit release, perhaps because the current development version is unstable, use

git checkout ET_2011_06

The tree will be very quickly updated to match the release. All changed files will have updated datestamps, so you should be able to trust the Cactus make system to recompile only what is necessary. However, it might be safer to delete any configurations and build them again.

==Committing==
For sub-repositories which are upstream (as most are), regardless of which revision control system is used upstream you must first initialise the sub-repository for committing:

git module init-upstream <submodule-path>

from the root Cactus (super-repo) directory. You can get a list of the available submodules with "git module ls" or by using tab completion if your shell is bash. Once you have completed this process for a submodule, the method of pushing depends on the version control system used upstream.

===Git===
For thorns whose source repositories are in Git already, you can commit and push as normal in the subrepositories.

git push upstream ...

===SVN===
For submodules which use SVN upstream, you can commit directly to the source SVN repository using, from the submodule directory,

git svn dcommit

assuming you have commit rights in the source SVN repository. This will push any local commits to SVN.

===Mercurial===

For Mercurial upstream repositories, it is more complicated, as there is no git-hg (equivalent to git-svn). In this case, you make commits locally to your submodule as if it were a regular git repository. Then, when you want to push upstream:

cd <submodule-path>.hg
hg pull git # Pull from the git submodule
hg push # Push to the upstream hg repo

This obviously has to be improved.

==Viewing history==

Each sub-repository is a separate fully-fledged Git repository, so you can go into each one and type "git status", "git log" etc. This works for repositories which are SVN upstream, because we use git-svn to convert the repositories to Git on the server. You can also use the usual graphical tools (we recommend GitX from [http://brotherbard.com/blog/2010/03/experimental-gitx-fork/] for Mac OS, and XXX for Linux) on each subrepository to visualise the log messages and patches, see which files have local changes, and interactively commit parts of files.

==Viewing local modifications==

You can see at a glance which of the submodules have local changes using

git status

You can then go into each submodule and see those changes using git status or git diff.

GitSuperRepoUsersGuide

2011-06-27T13:53:48Z

Barry.wardell: /* SVN */

[[Category:GitSuperRepo]]

''Include here simple user-friendly instructions for using a Cactus super-repository''

We provide a Git super-repo of the Einstein Toolkit. This page describes how to check it out and work with it. It is currently more of a tutorial than a User Guide.

Note that these instructions are a work in progress and will not currently work.

Much of this is too complicated to remember and understand. We need to make it much simpler, possibly by providing scripts.

==Checking out==

git clone --recursive cactus@git.einsteintoolkit.org:einsteintoolkit Cactus

This takes about 10 minutes and will create a directory called Cactus containing the entire Einstein Toolkit and all its version history. When it is done, setup the git-module package which provides convenience functionality:

cd Cactus
./bin/git-module setup && source ~/.profile

==Updating==

To update the checkout, it is first a good idea to see what would be changed. First enter the top-level super-repo directory. You can get a list of the changes using

git submodule foreach 'git --no-pager log --oneline ..remotes/origin/HEAD'

This assumes that the remote HEAD is the one that is checked out, which is not true for SimFactory, for example. Also, don't we need to do some fetching first?

If this is OK, you can then update using:

git pull
git submodule update

==Checking out a specific branch==

If you want to use a particular Einstein Toolkit release, perhaps because the current development version is unstable, use

git checkout ET_2011_06

The tree will be very quickly updated to match the release. All changed files will have updated datestamps, so you should be able to trust the Cactus make system to recompile only what is necessary. However, it might be safer to delete any configurations and build them again.

==Committing==
For sub-repositories which are upstream (as most are), regardless of which revision control system is used upstream you must first initialise the sub-repository for committing:

git module init-upstream <submodule-path>

from the root Cactus (super-repo) directory. You can get a list of the available submodules with "git module ls" or by using tab completion if your shell is bash. Once you have completed this process for a submodule, the method of pushing depends on the version control system used upstream.

===Git===
For thorns whose source repositories are in Git already, you can commit and push as normal in the subrepositories.

git push upstream ...

===SVN===
For submodules which use SVN upstream, you can commit directly to the source SVN repository using, from the submodule directory,

git svn dcommit

assuming you have commit rights in the source SVN repository. This will push any local commits to SVN.

===Mercurial===

For Mercurial upstream repositories, it is more complicated, as there is no git-hg (equivalent to git-svn). In this case, you make commits locally to your submodule as if it were a regular git repository. Then, when you want to push upstream:

cd <submodule-path>.hg
hg pull git # Pull from the git submodule
hg push # Push to the upstream hg repo

This obviously has to be improved.

==Viewing history==

Each sub-repository is a separate fully-fledged Git repository, so you can go into each one and type "git status", "git log" etc. This works for repositories which are SVN upstream, because we use git-svn to convert the repositories to Git on the server. You can also use the usual graphical tools (we recommend GitX from [http://brotherbard.com/blog/2010/03/experimental-gitx-fork/] for Mac OS, and XXX for Linux) on each subrepository to visualise the log messages and patches, see which files have local changes, and interactively commit parts of files.

==Viewing local modifications==

You can see at a glance which of the submodules have local changes using

git status

You can then go into each submodule and see those changes using git status or git diff.

GitSuperRepoUsersGuide

2011-06-27T13:53:01Z

Barry.wardell: /* Committing */

[[Category:GitSuperRepo]]

''Include here simple user-friendly instructions for using a Cactus super-repository''

We provide a Git super-repo of the Einstein Toolkit. This page describes how to check it out and work with it. It is currently more of a tutorial than a User Guide.

Note that these instructions are a work in progress and will not currently work.

Much of this is too complicated to remember and understand. We need to make it much simpler, possibly by providing scripts.

==Checking out==

git clone --recursive cactus@git.einsteintoolkit.org:einsteintoolkit Cactus

This takes about 10 minutes and will create a directory called Cactus containing the entire Einstein Toolkit and all its version history. When it is done, setup the git-module package which provides convenience functionality:

cd Cactus
./bin/git-module setup && source ~/.profile

==Updating==

To update the checkout, it is first a good idea to see what would be changed. First enter the top-level super-repo directory. You can get a list of the changes using

git submodule foreach 'git --no-pager log --oneline ..remotes/origin/HEAD'

This assumes that the remote HEAD is the one that is checked out, which is not true for SimFactory, for example. Also, don't we need to do some fetching first?

If this is OK, you can then update using:

git pull
git submodule update

==Checking out a specific branch==

If you want to use a particular Einstein Toolkit release, perhaps because the current development version is unstable, use

git checkout ET_2011_06

The tree will be very quickly updated to match the release. All changed files will have updated datestamps, so you should be able to trust the Cactus make system to recompile only what is necessary. However, it might be safer to delete any configurations and build them again.

==Committing==
For sub-repositories which are upstream (as most are), regardless of which revision control system is used upstream you must first initialise the sub-repository for committing:

git module init-upstream <submodule-path>

from the root Cactus (super-repo) directory. You can get a list of the available submodules with "git module ls" or by using tab completion if your shell is bash. Once you have completed this process for a submodule, the method of pushing depends on the version control system used upstream.

===Git===
For thorns whose source repositories are in Git already, you can commit and push as normal in the subrepositories.

git push upstream ...

===SVN===
You can now commit directly to the source SVN repository using, from the submodule directory,

git svn dcommit

assuming you have commit rights in the source SVN repository. This will push any local commits to SVN.

===Mercurial===

For Mercurial upstream repositories, it is more complicated, as there is no git-hg (equivalent to git-svn). In this case, you make commits locally to your submodule as if it were a regular git repository. Then, when you want to push upstream:

cd <submodule-path>.hg
hg pull git # Pull from the git submodule
hg push # Push to the upstream hg repo

This obviously has to be improved.

==Viewing history==

Each sub-repository is a separate fully-fledged Git repository, so you can go into each one and type "git status", "git log" etc. This works for repositories which are SVN upstream, because we use git-svn to convert the repositories to Git on the server. You can also use the usual graphical tools (we recommend GitX from [http://brotherbard.com/blog/2010/03/experimental-gitx-fork/] for Mac OS, and XXX for Linux) on each subrepository to visualise the log messages and patches, see which files have local changes, and interactively commit parts of files.

==Viewing local modifications==

You can see at a glance which of the submodules have local changes using

git status

You can then go into each submodule and see those changes using git status or git diff.

GitSuperRepoUsersGuide

2011-06-27T13:52:47Z

Barry.wardell: /* Committing */

[[Category:GitSuperRepo]]

''Include here simple user-friendly instructions for using a Cactus super-repository''

We provide a Git super-repo of the Einstein Toolkit. This page describes how to check it out and work with it. It is currently more of a tutorial than a User Guide.

Note that these instructions are a work in progress and will not currently work.

Much of this is too complicated to remember and understand. We need to make it much simpler, possibly by providing scripts.

==Checking out==

git clone --recursive cactus@git.einsteintoolkit.org:einsteintoolkit Cactus

This takes about 10 minutes and will create a directory called Cactus containing the entire Einstein Toolkit and all its version history. When it is done, setup the git-module package which provides convenience functionality:

cd Cactus
./bin/git-module setup && source ~/.profile

==Updating==

To update the checkout, it is first a good idea to see what would be changed. First enter the top-level super-repo directory. You can get a list of the changes using

git submodule foreach 'git --no-pager log --oneline ..remotes/origin/HEAD'

This assumes that the remote HEAD is the one that is checked out, which is not true for SimFactory, for example. Also, don't we need to do some fetching first?

If this is OK, you can then update using:

git pull
git submodule update

==Checking out a specific branch==

If you want to use a particular Einstein Toolkit release, perhaps because the current development version is unstable, use

git checkout ET_2011_06

The tree will be very quickly updated to match the release. All changed files will have updated datestamps, so you should be able to trust the Cactus make system to recompile only what is necessary. However, it might be safer to delete any configurations and build them again.

==Committing==
For sub-repositories which are upstream (as most are), regardless of which revision control system is used upstream you must first initialise the sub-repository for committing:

git module init-upstream <submodule-path>

from the root Cactus (super-repo) directory. You can get a list of the available submodules with "git module ls" or by using tab completion if your shell is bash. Once you have completed this process for a submodule, the method of pushing depends on the version control used upstream.

===Git===
For thorns whose source repositories are in Git already, you can commit and push as normal in the subrepositories.

git push upstream ...

===SVN===
You can now commit directly to the source SVN repository using, from the submodule directory,

git svn dcommit

assuming you have commit rights in the source SVN repository. This will push any local commits to SVN.

===Mercurial===

For Mercurial upstream repositories, it is more complicated, as there is no git-hg (equivalent to git-svn). In this case, you make commits locally to your submodule as if it were a regular git repository. Then, when you want to push upstream:

cd <submodule-path>.hg
hg pull git # Pull from the git submodule
hg push # Push to the upstream hg repo

This obviously has to be improved.

==Viewing history==

Each sub-repository is a separate fully-fledged Git repository, so you can go into each one and type "git status", "git log" etc. This works for repositories which are SVN upstream, because we use git-svn to convert the repositories to Git on the server. You can also use the usual graphical tools (we recommend GitX from [http://brotherbard.com/blog/2010/03/experimental-gitx-fork/] for Mac OS, and XXX for Linux) on each subrepository to visualise the log messages and patches, see which files have local changes, and interactively commit parts of files.

==Viewing local modifications==

You can see at a glance which of the submodules have local changes using

git status

You can then go into each submodule and see those changes using git status or git diff.

GitSuperRepoUsersGuide

2011-06-27T13:52:08Z

Barry.wardell: /* Git */

[[Category:GitSuperRepo]]

''Include here simple user-friendly instructions for using a Cactus super-repository''

We provide a Git super-repo of the Einstein Toolkit. This page describes how to check it out and work with it. It is currently more of a tutorial than a User Guide.

Note that these instructions are a work in progress and will not currently work.

Much of this is too complicated to remember and understand. We need to make it much simpler, possibly by providing scripts.

==Checking out==

git clone --recursive cactus@git.einsteintoolkit.org:einsteintoolkit Cactus

This takes about 10 minutes and will create a directory called Cactus containing the entire Einstein Toolkit and all its version history. When it is done, setup the git-module package which provides convenience functionality:

cd Cactus
./bin/git-module setup && source ~/.profile

==Updating==

To update the checkout, it is first a good idea to see what would be changed. First enter the top-level super-repo directory. You can get a list of the changes using

git submodule foreach 'git --no-pager log --oneline ..remotes/origin/HEAD'

This assumes that the remote HEAD is the one that is checked out, which is not true for SimFactory, for example. Also, don't we need to do some fetching first?

If this is OK, you can then update using:

git pull
git submodule update

==Checking out a specific branch==

If you want to use a particular Einstein Toolkit release, perhaps because the current development version is unstable, use

git checkout ET_2011_06

The tree will be very quickly updated to match the release. All changed files will have updated datestamps, so you should be able to trust the Cactus make system to recompile only what is necessary. However, it might be safer to delete any configurations and build them again.

==Committing==
For sub-repositories which are upstream (as most are), regardless of which revision control system is used upstream you must first initialise the sub-repository for committing:

git module init-upstream <submodule-path>

from the root Cactus (super-repo) directory. You can get a list of the available submodules with "git module ls" or by using tab completion if your shell is bash.

===Git===
For thorns whose source repositories are in Git already, you can commit and push as normal in the subrepositories.

git push upstream ...

===SVN===
You can now commit directly to the source SVN repository using, from the submodule directory,

git svn dcommit

assuming you have commit rights in the source SVN repository. This will push any local commits to SVN.

===Mercurial===

For Mercurial upstream repositories, it is more complicated, as there is no git-hg (equivalent to git-svn). In this case, you make commits locally to your submodule as if it were a regular git repository. Then, when you want to push upstream:

cd <submodule-path>.hg
hg pull git # Pull from the git submodule
hg push # Push to the upstream hg repo

This obviously has to be improved.

==Viewing history==

Each sub-repository is a separate fully-fledged Git repository, so you can go into each one and type "git status", "git log" etc. This works for repositories which are SVN upstream, because we use git-svn to convert the repositories to Git on the server. You can also use the usual graphical tools (we recommend GitX from [http://brotherbard.com/blog/2010/03/experimental-gitx-fork/] for Mac OS, and XXX for Linux) on each subrepository to visualise the log messages and patches, see which files have local changes, and interactively commit parts of files.

==Viewing local modifications==

You can see at a glance which of the submodules have local changes using

git status

You can then go into each submodule and see those changes using git status or git diff.

GitSuperRepoUsersGuide

2011-06-27T13:51:31Z

Barry.wardell: /* Git */

[[Category:GitSuperRepo]]

''Include here simple user-friendly instructions for using a Cactus super-repository''

We provide a Git super-repo of the Einstein Toolkit. This page describes how to check it out and work with it. It is currently more of a tutorial than a User Guide.

Note that these instructions are a work in progress and will not currently work.

Much of this is too complicated to remember and understand. We need to make it much simpler, possibly by providing scripts.

==Checking out==

git clone --recursive cactus@git.einsteintoolkit.org:einsteintoolkit Cactus

This takes about 10 minutes and will create a directory called Cactus containing the entire Einstein Toolkit and all its version history. When it is done, setup the git-module package which provides convenience functionality:

cd Cactus
./bin/git-module setup && source ~/.profile

==Updating==

To update the checkout, it is first a good idea to see what would be changed. First enter the top-level super-repo directory. You can get a list of the changes using

git submodule foreach 'git --no-pager log --oneline ..remotes/origin/HEAD'

This assumes that the remote HEAD is the one that is checked out, which is not true for SimFactory, for example. Also, don't we need to do some fetching first?

If this is OK, you can then update using:

git pull
git submodule update

==Checking out a specific branch==

If you want to use a particular Einstein Toolkit release, perhaps because the current development version is unstable, use

git checkout ET_2011_06

The tree will be very quickly updated to match the release. All changed files will have updated datestamps, so you should be able to trust the Cactus make system to recompile only what is necessary. However, it might be safer to delete any configurations and build them again.

==Committing==
For sub-repositories which are upstream (as most are), regardless of which revision control system is used upstream you must first initialise the sub-repository for committing:

git module init-upstream <submodule-path>

from the root Cactus (super-repo) directory. You can get a list of the available submodules with "git module ls" or by using tab completion if your shell is bash.

===Git===
For thorns whose source repositories are in Git already, once you have enabled upstream pushing you can commit and push as normal in the subrepositories.

git push upstream ...

===SVN===
You can now commit directly to the source SVN repository using, from the submodule directory,

git svn dcommit

assuming you have commit rights in the source SVN repository. This will push any local commits to SVN.

===Mercurial===

For Mercurial upstream repositories, it is more complicated, as there is no git-hg (equivalent to git-svn). In this case, you make commits locally to your submodule as if it were a regular git repository. Then, when you want to push upstream:

cd <submodule-path>.hg
hg pull git # Pull from the git submodule
hg push # Push to the upstream hg repo

This obviously has to be improved.

==Viewing history==

Each sub-repository is a separate fully-fledged Git repository, so you can go into each one and type "git status", "git log" etc. This works for repositories which are SVN upstream, because we use git-svn to convert the repositories to Git on the server. You can also use the usual graphical tools (we recommend GitX from [http://brotherbard.com/blog/2010/03/experimental-gitx-fork/] for Mac OS, and XXX for Linux) on each subrepository to visualise the log messages and patches, see which files have local changes, and interactively commit parts of files.

==Viewing local modifications==

You can see at a glance which of the submodules have local changes using

git status

You can then go into each submodule and see those changes using git status or git diff.

GitSuperRepoUsersGuide

2011-06-27T13:49:51Z

Barry.wardell: /* Committing */

[[Category:GitSuperRepo]]

''Include here simple user-friendly instructions for using a Cactus super-repository''

We provide a Git super-repo of the Einstein Toolkit. This page describes how to check it out and work with it. It is currently more of a tutorial than a User Guide.

Note that these instructions are a work in progress and will not currently work.

Much of this is too complicated to remember and understand. We need to make it much simpler, possibly by providing scripts.

==Checking out==

git clone --recursive cactus@git.einsteintoolkit.org:einsteintoolkit Cactus

This takes about 10 minutes and will create a directory called Cactus containing the entire Einstein Toolkit and all its version history. When it is done, setup the git-module package which provides convenience functionality:

cd Cactus
./bin/git-module setup && source ~/.profile

==Updating==

To update the checkout, it is first a good idea to see what would be changed. First enter the top-level super-repo directory. You can get a list of the changes using

git submodule foreach 'git --no-pager log --oneline ..remotes/origin/HEAD'

This assumes that the remote HEAD is the one that is checked out, which is not true for SimFactory, for example. Also, don't we need to do some fetching first?

If this is OK, you can then update using:

git pull
git submodule update

==Checking out a specific branch==

If you want to use a particular Einstein Toolkit release, perhaps because the current development version is unstable, use

git checkout ET_2011_06

The tree will be very quickly updated to match the release. All changed files will have updated datestamps, so you should be able to trust the Cactus make system to recompile only what is necessary. However, it might be safer to delete any configurations and build them again.

==Committing==
For sub-repositories which are upstream (as most are), regardless of which revision control system is used upstream you must first initialise the sub-repository for committing:

git module init-upstream <submodule-path>

from the root Cactus (super-repo) directory. You can get a list of the available submodules with "git module ls" or by using tab completion if your shell is bash.

===Git===
For thorns whose source repositories are in Git already, you can commit and push as normal in the subrepositories.

git push upstream ...

===SVN===
You can now commit directly to the source SVN repository using, from the submodule directory,

git svn dcommit

assuming you have commit rights in the source SVN repository. This will push any local commits to SVN.

===Mercurial===

For Mercurial upstream repositories, it is more complicated, as there is no git-hg (equivalent to git-svn). In this case, you make commits locally to your submodule as if it were a regular git repository. Then, when you want to push upstream:

cd <submodule-path>.hg
hg pull git # Pull from the git submodule
hg push # Push to the upstream hg repo

This obviously has to be improved.

==Viewing history==

Each sub-repository is a separate fully-fledged Git repository, so you can go into each one and type "git status", "git log" etc. This works for repositories which are SVN upstream, because we use git-svn to convert the repositories to Git on the server. You can also use the usual graphical tools (we recommend GitX from [http://brotherbard.com/blog/2010/03/experimental-gitx-fork/] for Mac OS, and XXX for Linux) on each subrepository to visualise the log messages and patches, see which files have local changes, and interactively commit parts of files.

==Viewing local modifications==

You can see at a glance which of the submodules have local changes using

git status

You can then go into each submodule and see those changes using git status or git diff.

GitSuperRepoUsersGuide

2011-06-27T13:49:16Z

Barry.wardell: /* Committing */

[[Category:GitSuperRepo]]

''Include here simple user-friendly instructions for using a Cactus super-repository''

We provide a Git super-repo of the Einstein Toolkit. This page describes how to check it out and work with it. It is currently more of a tutorial than a User Guide.

Note that these instructions are a work in progress and will not currently work.

Much of this is too complicated to remember and understand. We need to make it much simpler, possibly by providing scripts.

==Checking out==

git clone --recursive cactus@git.einsteintoolkit.org:einsteintoolkit Cactus

This takes about 10 minutes and will create a directory called Cactus containing the entire Einstein Toolkit and all its version history. When it is done, setup the git-module package which provides convenience functionality:

cd Cactus
./bin/git-module setup && source ~/.profile

==Updating==

To update the checkout, it is first a good idea to see what would be changed. First enter the top-level super-repo directory. You can get a list of the changes using

git submodule foreach 'git --no-pager log --oneline ..remotes/origin/HEAD'

This assumes that the remote HEAD is the one that is checked out, which is not true for SimFactory, for example. Also, don't we need to do some fetching first?

If this is OK, you can then update using:

git pull
git submodule update

==Checking out a specific branch==

If you want to use a particular Einstein Toolkit release, perhaps because the current development version is unstable, use

git checkout ET_2011_06

The tree will be very quickly updated to match the release. All changed files will have updated datestamps, so you should be able to trust the Cactus make system to recompile only what is necessary. However, it might be safer to delete any configurations and build them again.

==Committing==
For sub-repositories which are upstream (as most are), regardless of which revision control system is used upstream you must first initialise the sub-repository for committing:

git module init-upstream <submodule-path>

e.g.

git module init-upstream simfactory

from the root Cactus (super-repo) directory. You can get a list of the available submodules with "git module ls" or by using tab completion if your shell is bash.

===Git===
For thorns whose source repositories are in Git already, you can commit and push as normal in the subrepositories.

git push upstream ...

===SVN===
You can now commit directly to the source SVN repository using, from the submodule directory,

git svn dcommit

assuming you have commit rights in the source SVN repository. This will push any local commits to SVN.

===Mercurial===

For Mercurial upstream repositories, it is more complicated, as there is no git-hg (equivalent to git-svn). In this case, you make commits locally to your submodule as if it were a regular git repository. Then, when you want to push upstream:

cd <submodule-path>.hg
hg pull git # Pull from the git submodule
hg push # Push to the upstream hg repo

This obviously has to be improved.

==Viewing history==

Each sub-repository is a separate fully-fledged Git repository, so you can go into each one and type "git status", "git log" etc. This works for repositories which are SVN upstream, because we use git-svn to convert the repositories to Git on the server. You can also use the usual graphical tools (we recommend GitX from [http://brotherbard.com/blog/2010/03/experimental-gitx-fork/] for Mac OS, and XXX for Linux) on each subrepository to visualise the log messages and patches, see which files have local changes, and interactively commit parts of files.

==Viewing local modifications==

You can see at a glance which of the submodules have local changes using

git status

You can then go into each submodule and see those changes using git status or git diff.

GitSuperRepoUsersGuide

2011-06-27T13:49:00Z

Barry.wardell: /* Committing */

[[Category:GitSuperRepo]]

''Include here simple user-friendly instructions for using a Cactus super-repository''

We provide a Git super-repo of the Einstein Toolkit. This page describes how to check it out and work with it. It is currently more of a tutorial than a User Guide.

Note that these instructions are a work in progress and will not currently work.

Much of this is too complicated to remember and understand. We need to make it much simpler, possibly by providing scripts.

==Checking out==

git clone --recursive cactus@git.einsteintoolkit.org:einsteintoolkit Cactus

This takes about 10 minutes and will create a directory called Cactus containing the entire Einstein Toolkit and all its version history. When it is done, setup the git-module package which provides convenience functionality:

cd Cactus
./bin/git-module setup && source ~/.profile

==Updating==

To update the checkout, it is first a good idea to see what would be changed. First enter the top-level super-repo directory. You can get a list of the changes using

git submodule foreach 'git --no-pager log --oneline ..remotes/origin/HEAD'

This assumes that the remote HEAD is the one that is checked out, which is not true for SimFactory, for example. Also, don't we need to do some fetching first?

If this is OK, you can then update using:

git pull
git submodule update

==Checking out a specific branch==

If you want to use a particular Einstein Toolkit release, perhaps because the current development version is unstable, use

git checkout ET_2011_06

The tree will be very quickly updated to match the release. All changed files will have updated datestamps, so you should be able to trust the Cactus make system to recompile only what is necessary. However, it might be safer to delete any configurations and build them again.

==Committing==
For sub-repositories which are upstream (as most are), regardless of which revision control system is used upstream you must first initialise the sub-repository for committing:

git module init-upstream <submodule-path>

e.g.

git module init-upstream simfactory

from the root Cactus (super-repo) directory. You can get a list of the available submodules with "git module ls" or by using tab completion if your shell is bash.

===Git===
For thorns whose source repositories are in Git already, you can commit and push as normal in the subrepositories.

git push upstream ...

===SVN===
You can now commit directly to the source SVN repository using, from the submodule directory,

git svn dcommit

assuming you have commit rights in the source SVN repository. This will push any local commits to SVN.

===Mercurial===

For Mercurial upstream repositories, it is more complicated, as there is no git-hg (equivalent to git-svn). In this case, you make commits locally to your submodule as if it were a regular git repository. Then, when you want to push upstream:

cd <submodule-path>.hg
hg pull git # Pull from the git submodule
hg push # Push to the upstream hg repo

This obviously has to be improved.

==Viewing history==

Each sub-repository is a separate fully-fledged Git repository, so you can go into each one and type "git status", "git log" etc. This works for repositories which are SVN upstream, because we use git-svn to convert the repositories to Git on the server. You can also use the usual graphical tools (we recommend GitX from [http://brotherbard.com/blog/2010/03/experimental-gitx-fork/] for Mac OS, and XXX for Linux) on each subrepository to visualise the log messages and patches, see which files have local changes, and interactively commit parts of files.

==Viewing local modifications==

You can see at a glance which of the submodules have local changes using

git status

You can then go into each submodule and see those changes using git status or git diff.

GitSuperRepoUsersGuide

2011-06-27T08:40:26Z

Barry.wardell: /* Checking out */

[[Category:GitSuperRepo]]

''Include here simple user-friendly instructions for using a Cactus super-repository''

We provide a Git super-repo of the Einstein Toolkit. This page describes how to check it out and work with it. It is currently more of a tutorial than a User Guide.

Note that these instructions are a work in progress and will not currently work.

Much of this is too complicated to remember and understand. We need to make it much simpler, possibly by providing scripts.

==Checking out==

git clone --recursive cactus@git.einsteintoolkit.org:einsteintoolkit Cactus

This takes about 10 minutes and will create a directory called Cactus containing the entire Einstein Toolkit and all its version history. When it is done, setup the git-module package which provides convenience functionality:

cd Cactus
./bin/git-module setup && source ~/.profile

==Updating==

To update the checkout, it is first a good idea to see what would be changed. First enter the top-level super-repo directory. You can get a list of the changes using

git submodule foreach 'git --no-pager log --oneline ..remotes/origin/HEAD'

This assumes that the remote HEAD is the one that is checked out, which is not true for SimFactory, for example.

If this is OK, you can then update using:

git pull
git submodule update

==Checking out a specific branch==

If you want to use a particular Einstein Toolkit release, perhaps because the current development version is unstable, use

git checkout ET_2011_06

The tree will be very quickly updated to match the release. All changed files will have updated datestamps, so you should be able to trust the Cactus make system to recompile only what is necessary. However, it might be safer to delete any configurations and build them again.

==Committing==

===Git===

For thorns whose source repositories are in Git already, you can commit and push as normal in the subrepositories. If they have been cloned from a public URL, as for the Einstein Toolkit URL above, you can use

cd <submodule-path>
git remote add upstream <new-url>
git push upstream ...

===SVN===

For subrepositories which are SVN upstream (for example simfactory), you must first initialise the subrepository for committing:

cd simfactory
git checkout trunk
git svn init -s --prefix=origin/ svn://...

You can now commit directly to the source SVN repository using

git svn dcommit ...

assuming you have commit rights in the source SVN repository.

===Mercurial===

For Mercurial upstream repositories, it is more complicated, as there is no git-hg (equivalent to git-svn):

hg clone <hg-url> <submodule-path>-upstream
cd <submodule-path>-upstream
vi .hg/hgrc # Add:
# [git]
# intree = 1
# [path]
# git = ../<submodule-path>
hg bookmark master -r default
hg gexport # Can take quite a long time (~1 hour for carpet-hg)
hg pull git # Pull from the git submodule
hg push # Push to the upstream hg repo

This obviously has to be improved.

==Viewing history==

Each sub-repository is a separate fully-fledged Git repository, so you can go into each one and type "git status", "git log" etc. This works for repositories which are SVN upstream, because we use git-svn to convert the repositories to Git on the server. You can also use the usual graphical tools (we recommend GitX from [http://brotherbard.com/blog/2010/03/experimental-gitx-fork/] for Mac OS, and XXX for Linux) on each subrepository to visualise the log messages and patches, see which files have local changes, and interactively commit parts of files.

==Viewing local modifications==

You can see at a glance which of the submodules have local changes using

git status

You can then go into each submodule and see those changes using git status or git diff.

GitSuperRepoUsersGuide

2011-06-27T08:40:06Z

Barry.wardell: /* Checking out */

[[Category:GitSuperRepo]]

''Include here simple user-friendly instructions for using a Cactus super-repository''

We provide a Git super-repo of the Einstein Toolkit. This page describes how to check it out and work with it. It is currently more of a tutorial than a User Guide.

Note that these instructions are a work in progress and will not currently work.

Much of this is too complicated to remember and understand. We need to make it much simpler, possibly by providing scripts.

==Checking out==

git clone --recursive cactus@git.einsteintoolkit.org:einsteintoolkit Cactus

This takes about 10 minutes and will create a directory called Cactus containing the entire Einstein Toolkit and all its version history. When it is done, setup the git-module package which provides convenience functionality:

cd Cactus
./bin/git-module setup
. ~/.profile

==Updating==

To update the checkout, it is first a good idea to see what would be changed. First enter the top-level super-repo directory. You can get a list of the changes using

git submodule foreach 'git --no-pager log --oneline ..remotes/origin/HEAD'

This assumes that the remote HEAD is the one that is checked out, which is not true for SimFactory, for example.

If this is OK, you can then update using:

git pull
git submodule update

==Checking out a specific branch==

If you want to use a particular Einstein Toolkit release, perhaps because the current development version is unstable, use

git checkout ET_2011_06

The tree will be very quickly updated to match the release. All changed files will have updated datestamps, so you should be able to trust the Cactus make system to recompile only what is necessary. However, it might be safer to delete any configurations and build them again.

==Committing==

===Git===

For thorns whose source repositories are in Git already, you can commit and push as normal in the subrepositories. If they have been cloned from a public URL, as for the Einstein Toolkit URL above, you can use

cd <submodule-path>
git remote add upstream <new-url>
git push upstream ...

===SVN===

For subrepositories which are SVN upstream (for example simfactory), you must first initialise the subrepository for committing:

cd simfactory
git checkout trunk
git svn init -s --prefix=origin/ svn://...

You can now commit directly to the source SVN repository using

git svn dcommit ...

assuming you have commit rights in the source SVN repository.

===Mercurial===

For Mercurial upstream repositories, it is more complicated, as there is no git-hg (equivalent to git-svn):

hg clone <hg-url> <submodule-path>-upstream
cd <submodule-path>-upstream
vi .hg/hgrc # Add:
# [git]
# intree = 1
# [path]
# git = ../<submodule-path>
hg bookmark master -r default
hg gexport # Can take quite a long time (~1 hour for carpet-hg)
hg pull git # Pull from the git submodule
hg push # Push to the upstream hg repo

This obviously has to be improved.

==Viewing history==

Each sub-repository is a separate fully-fledged Git repository, so you can go into each one and type "git status", "git log" etc. This works for repositories which are SVN upstream, because we use git-svn to convert the repositories to Git on the server. You can also use the usual graphical tools (we recommend GitX from [http://brotherbard.com/blog/2010/03/experimental-gitx-fork/] for Mac OS, and XXX for Linux) on each subrepository to visualise the log messages and patches, see which files have local changes, and interactively commit parts of files.

==Viewing local modifications==

You can see at a glance which of the submodules have local changes using

git status

You can then go into each submodule and see those changes using git status or git diff.

GitSuperRepoUsersGuide

2011-06-27T08:39:46Z

Barry.wardell: /* Checking out */

[[Category:GitSuperRepo]]

''Include here simple user-friendly instructions for using a Cactus super-repository''

We provide a Git super-repo of the Einstein Toolkit. This page describes how to check it out and work with it. It is currently more of a tutorial than a User Guide.

Note that these instructions are a work in progress and will not currently work.

Much of this is too complicated to remember and understand. We need to make it much simpler, possibly by providing scripts.

==Checking out==

git clone --recursive cactus@git.einsteintoolkit.org:einsteintoolkit Cactus

This takes about 10 minutes and will create a directory called Cactus containing the entire Einstein Toolkit and all its version history. When it is done, setup the git-module package which provides convenience functionality:

cd Cactus
./bin/git-module setup

==Updating==

To update the checkout, it is first a good idea to see what would be changed. First enter the top-level super-repo directory. You can get a list of the changes using

git submodule foreach 'git --no-pager log --oneline ..remotes/origin/HEAD'

This assumes that the remote HEAD is the one that is checked out, which is not true for SimFactory, for example.

If this is OK, you can then update using:

git pull
git submodule update

==Checking out a specific branch==

If you want to use a particular Einstein Toolkit release, perhaps because the current development version is unstable, use

git checkout ET_2011_06

The tree will be very quickly updated to match the release. All changed files will have updated datestamps, so you should be able to trust the Cactus make system to recompile only what is necessary. However, it might be safer to delete any configurations and build them again.

==Committing==

===Git===

For thorns whose source repositories are in Git already, you can commit and push as normal in the subrepositories. If they have been cloned from a public URL, as for the Einstein Toolkit URL above, you can use

cd <submodule-path>
git remote add upstream <new-url>
git push upstream ...

===SVN===

For subrepositories which are SVN upstream (for example simfactory), you must first initialise the subrepository for committing:

cd simfactory
git checkout trunk
git svn init -s --prefix=origin/ svn://...

You can now commit directly to the source SVN repository using

git svn dcommit ...

assuming you have commit rights in the source SVN repository.

===Mercurial===

For Mercurial upstream repositories, it is more complicated, as there is no git-hg (equivalent to git-svn):

hg clone <hg-url> <submodule-path>-upstream
cd <submodule-path>-upstream
vi .hg/hgrc # Add:
# [git]
# intree = 1
# [path]
# git = ../<submodule-path>
hg bookmark master -r default
hg gexport # Can take quite a long time (~1 hour for carpet-hg)
hg pull git # Pull from the git submodule
hg push # Push to the upstream hg repo

This obviously has to be improved.

==Viewing history==

Each sub-repository is a separate fully-fledged Git repository, so you can go into each one and type "git status", "git log" etc. This works for repositories which are SVN upstream, because we use git-svn to convert the repositories to Git on the server. You can also use the usual graphical tools (we recommend GitX from [http://brotherbard.com/blog/2010/03/experimental-gitx-fork/] for Mac OS, and XXX for Linux) on each subrepository to visualise the log messages and patches, see which files have local changes, and interactively commit parts of files.

==Viewing local modifications==

You can see at a glance which of the submodules have local changes using

git status

You can then go into each submodule and see those changes using git status or git diff.

GitSuperRepoUsersGuide

2011-06-27T08:39:29Z

Barry.wardell: /* Checking out */

[[Category:GitSuperRepo]]

''Include here simple user-friendly instructions for using a Cactus super-repository''

We provide a Git super-repo of the Einstein Toolkit. This page describes how to check it out and work with it. It is currently more of a tutorial than a User Guide.

Note that these instructions are a work in progress and will not currently work.

Much of this is too complicated to remember and understand. We need to make it much simpler, possibly by providing scripts.

==Checking out==

git clone --recursive cactus@git.einsteintoolkit.org:einsteintoolkit Cactus

This takes about 10 minutes and will create a directory called Cactus containing the entire Einstein Toolkit and all its version history. When it is done, setup the git-module package which provides convenience functionality:

cd Cactus
bin/git-module setup

==Updating==

To update the checkout, it is first a good idea to see what would be changed. First enter the top-level super-repo directory. You can get a list of the changes using

git submodule foreach 'git --no-pager log --oneline ..remotes/origin/HEAD'

This assumes that the remote HEAD is the one that is checked out, which is not true for SimFactory, for example.

If this is OK, you can then update using:

git pull
git submodule update

==Checking out a specific branch==

If you want to use a particular Einstein Toolkit release, perhaps because the current development version is unstable, use

git checkout ET_2011_06

The tree will be very quickly updated to match the release. All changed files will have updated datestamps, so you should be able to trust the Cactus make system to recompile only what is necessary. However, it might be safer to delete any configurations and build them again.

==Committing==

===Git===

For thorns whose source repositories are in Git already, you can commit and push as normal in the subrepositories. If they have been cloned from a public URL, as for the Einstein Toolkit URL above, you can use

cd <submodule-path>
git remote add upstream <new-url>
git push upstream ...

===SVN===

For subrepositories which are SVN upstream (for example simfactory), you must first initialise the subrepository for committing:

cd simfactory
git checkout trunk
git svn init -s --prefix=origin/ svn://...

You can now commit directly to the source SVN repository using

git svn dcommit ...

assuming you have commit rights in the source SVN repository.

===Mercurial===

For Mercurial upstream repositories, it is more complicated, as there is no git-hg (equivalent to git-svn):

hg clone <hg-url> <submodule-path>-upstream
cd <submodule-path>-upstream
vi .hg/hgrc # Add:
# [git]
# intree = 1
# [path]
# git = ../<submodule-path>
hg bookmark master -r default
hg gexport # Can take quite a long time (~1 hour for carpet-hg)
hg pull git # Pull from the git submodule
hg push # Push to the upstream hg repo

This obviously has to be improved.

==Viewing history==

Each sub-repository is a separate fully-fledged Git repository, so you can go into each one and type "git status", "git log" etc. This works for repositories which are SVN upstream, because we use git-svn to convert the repositories to Git on the server. You can also use the usual graphical tools (we recommend GitX from [http://brotherbard.com/blog/2010/03/experimental-gitx-fork/] for Mac OS, and XXX for Linux) on each subrepository to visualise the log messages and patches, see which files have local changes, and interactively commit parts of files.

==Viewing local modifications==

You can see at a glance which of the submodules have local changes using

git status

You can then go into each submodule and see those changes using git status or git diff.

GitSuperRepoUnsorted

2011-06-27T08:36:00Z

Barry.wardell:

[[Category:GitSuperRepo]]

''This is a list of things that would be useful to include in the documentation somewhere''

* A possible way to link the problems and solutions would be with two or three common scenarios. These would highlight a limitation/annoyance and show how our system provides an easy solution.

* A nice summary: The idea is to have a central "group" repository which points to a series of git and git-svn repositories as submodules, checked out using GetComponents. The group repository is what is cloned by users in the group, and will contain both public and private code. Gitolite can handle permissions based on users and branches. The group repository (and its submodules) would be updated from the upstream versions on any commits, and users could pull from the group repository when they wanted to.

* We could also have branches in the repository for the ET releases, and switching between them should be a lot easier than it is now (which is usually a case of "nuke your tree and check out again").

* We should also provide a public example of such a repository, containing e.g. the Einstein Toolkit. This could then be used in our ET demonstration.

==Current setup==

* A git repository which only contains submodules and a few symlinks.
* Git submodules for each remote repository.
* In the case where the remote repository is an svn server, there is a git-svn clone hosted by us. Users can create and push new branches to this clone (provided they're named numrel/*), but cannot update the branches which come from svn.
* Similarly for remote Mercurial repositories I have a git clone which we host.
* For remote git repositories where we may want 'internal' branches, we host a mirror of the remote repository and allow additional branches to be created. These additional branches are only on our mirror, they are not pushed back upstream. As an example of this, we have a mirror of McLachlan which has a branch with vectorisation enabled.
* All mirrors (git-svn, git-hg and git) are automatically kept up to date with an hourly cron job, as is the 'submodules' branch of the super-repository.
* Commiting back upstream works for git, svn and hg.

GitSuperRepoAdminGuide

2011-06-27T08:35:49Z

Barry.wardell:

[[Category:GitSuperRepo]]

''Include here a guide to setting up a Git server with a Cactus super-repository''

Have temporarily copy+pasted the README which is in note form - need to write a proper up-to-date guide.

<pre>

###############################################################################
# CactusGit
###############################################################################

# Introduction

# This package allows you to check out Cactus using GetComponents from
# a CRL thornlist using git-svn instead of svn for the svn
# repositories. This means that your local Cactus tree contains
# complete version information and history and can be managed using
# git rather than svn.

# Further, the package provides tools for creating a
# "super-repository" for the tree with pointers to the downloaded
# repositories as git submodules. This means that the state of the
# tree can be identified by a single commit ID and can be cloned
# directly without having to use GetComponents or git-svn.

# Notes
# * To make this more convenient, small wrapper scripts can be written
# to replace the recipes given below
#
# * GetComponents could be extended to support git-svn directly,
# rather than using the wrapper-script approach that is currently
# used

# Tutorial

# Add our tools to the path, including the svn wrapper which tricks
# GetComponents into making a git-svn clone of each svn repository.
# We also include a copy of GetComponents here for convenience.
export OLDPATH=$PATH PATH=$PWD/bin:$PATH

# Check out the tree into the Cactus directory (can take a very long
# time, ~60 minutes, and does not show progress for each repository
# until it has checked it out completely)
GetComponents -v -a WaveToy.th

# Now manually checkout any thorns which require authentication
git svn clone ...

# Add the git-svn repos to your server (also add the Cactus super-repo).
crlrepos WaveToy.th > gitsvn.list
cd <gitolite_dir>
vi conf/gitolite.conf # Add the repositories in gitsvn.list
git commit conf/gitolite.conf
git push

# Add a 'fetching and pushing' (fp) user
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

# Push as the fp user
scp Cactus fp@server:fetch/
scp gitsvn.list fp@server:fetch/
ssh fp@server
cd fetch
for i in `cat gitsvn.list`; do cd $i && git remote add bare cactus@localhost:$i && git config --unset remote.bare.fetch && git config remote.bare.push 'refs/remotes/*:refs/heads/*' && git push bare && cd -; done

# On the server, update the HEAD of the git-svn repos
ssh cactus@server
cd ~/repositories
for i in `cat ../fetch/gitsvn`; do cd $i.git && git symbolic-ref HEAD refs/heads/git-svn && cd -; done

# Generate a super-repository called CactusGit using all the
# repositories from Cactus listed in WaveToy.th
crltogit CactusGit cactus@server: Llama.CTGamma.ET.th

# Add non-svn repos
GetComponents Llama.CTGamma.ET.th
rm -rf repos
git submodule add ... repos/...

# Symlink flesh
ln -s flesh/CONTRIBUTORS
ln -s flesh/COPYRIGHT
ln -s flesh/Makefile
ln -s flesh/doc
ln -s flesh/lib

# Commit symlinks (as fp)
git add *
git commit ...
git push cactus@localhost:Cactus master:master

######### Administrators ########
# Updating the git-svn mirrors
for i in `cat gitsvn.list`; do cd $i; git svn fetch && git push bare; cd -; done

# Updating the hg-git mirrors
for i in `cat githg.list`; do cd $i; hg pull && hg push git; cd -; done

# Updating the git mirrors
for i in `cat git.list`; do cd $i; git fetch -a -u && git push bare --all; cd -; done

# Updating all submodules in the super-repo
cd Cactus
git pull
git submodule foreach git pull
git commit -m "Update submodules." -a

# Updating a single submodule in the super-repo
cd Cactus
cd <repo-path>
git pull
cd -
git commit -m "Update submodule <repo-path>." <repo-path>

# Automatic updates of super-repo
git clone --recursive cactus@localhost:Cactus
cd Cactus
git submodule foreach 'git remote set-url origin /var/cactus/repositories/$path cactus@git.barrywardell.net:/$path; echo OK'
git remote set-url origin /var/cactus/repositories/Cactus
git remote set-url --push origin cactus@git.barrywardell.net:Cactus
git checkout submodules
git pull origin submodules
git submodule foreach 'git fetch origin && git checkout origin'
git commit -m "Update submodules." -a
git push origin submodules

# Adding a git-svn repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi gitsvn.list # add <newrepo>
git svn clone -s <url> <newrepo>
cd <newrepo>
git remote add bare cactus@localhost:<newrepo>
git config --unset remote.bare.fetch
git config remote.bare.push 'refs/remotes/*:refs/heads/*'
git push bare
cd ~/repositories/<newrepo>
git symbolic-ref HEAD refs/heads/trunk
cd ~/fetch/<newrepo>
git pull bare

# Adding a hg-git repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi githg.list # add <newrepo>
hg clone <url> <newrepo>
cd <newrepo>
vi .hg/hgrc # Add:
# [path]
# git = git+ssh://localhost/<newrepo>
# [git]
# intree = 1
hg gexport # Can take quite a long time
hg push git

# Adding a git mirror repo
vi conf/gitolite.conf
git commit conf/gitolite.conf
git push

ssh fp@server
cd fetch
vi git.list # add <newrepo>
git clone --mirror <url> <newrepo>
cd <newrepo>
git remote add bare cactus@localhost:<newrepo>
git config --unset remote.bare.fetch
git push bare --all

# Adding a submodule to the super-repo
git submodule add <url> <path>
git commit
git push

# Removing a submodule from the super-repo
vi .gitmodules
vi .git/config
git rm --cached <path>
git commit ...
git push

# Changing svn URL for a git-svn clone. WARNING: this is rewriting the history of
# the git-svn branch of your mirror and will mess peoples checkout up.
cd fetch/<repo>
git status
git gc
git filter-branch --msg-filter 'sed "s/git-svn-id: http/git-svn-id: https/g"' $(cat .git/packed-refs | awk '// {print $2}' | grep -v 'pack-refs')
sed -i -e "s/http/https/g" .git/config
rm -rf .git/svn
git svn rebase
git push bare --force

# Changing svn URL without rewriting history (https://git.wiki.kernel.org/index.php/GitSvnSwitch):
Edit the svn-remote url URL in .git/config to point to the new domain name
Run git svn fetch - This needs to fetch at least one new revision from svn!
Change svn-remote url back to the original url
Run git svn rebase -l to do a local rebase (with the changes that came in with the last fetch operation)
Change svn-remote url back to the new url
Run git svn rebase should now work again!

</pre>

GitSuperRepoUsersGuide

2011-06-27T08:35:39Z

Barry.wardell:

[[Category:GitSuperRepo]]

''Include here simple user-friendly instructions for using a Cactus super-repository''

We provide a Git super-repo of the Einstein Toolkit. This page describes how to check it out and work with it. It is currently more of a tutorial than a User Guide.

Note that these instructions are a work in progress and will not currently work.

Much of this is too complicated to remember and understand. We need to make it much simpler, possibly by providing scripts.

==Checking out==

git clone --recursive cactus@git.einsteintoolkit.org:einsteintoolkit Cactus

This takes about 10 minutes and will create a directory called Cactus containing the entire Einstein Toolkit and all its version history.

==Updating==

To update the checkout, it is first a good idea to see what would be changed. First enter the top-level super-repo directory. You can get a list of the changes using

git submodule foreach 'git --no-pager log --oneline ..remotes/origin/HEAD'

This assumes that the remote HEAD is the one that is checked out, which is not true for SimFactory, for example.

If this is OK, you can then update using:

git pull
git submodule update

==Checking out a specific branch==

If you want to use a particular Einstein Toolkit release, perhaps because the current development version is unstable, use

git checkout ET_2011_06

The tree will be very quickly updated to match the release. All changed files will have updated datestamps, so you should be able to trust the Cactus make system to recompile only what is necessary. However, it might be safer to delete any configurations and build them again.

==Committing==

===Git===

For thorns whose source repositories are in Git already, you can commit and push as normal in the subrepositories. If they have been cloned from a public URL, as for the Einstein Toolkit URL above, you can use

cd <submodule-path>
git remote add upstream <new-url>
git push upstream ...

===SVN===

For subrepositories which are SVN upstream (for example simfactory), you must first initialise the subrepository for committing:

cd simfactory
git checkout trunk
git svn init -s --prefix=origin/ svn://...

You can now commit directly to the source SVN repository using

git svn dcommit ...

assuming you have commit rights in the source SVN repository.

===Mercurial===

For Mercurial upstream repositories, it is more complicated, as there is no git-hg (equivalent to git-svn):

hg clone <hg-url> <submodule-path>-upstream
cd <submodule-path>-upstream
vi .hg/hgrc # Add:
# [git]
# intree = 1
# [path]
# git = ../<submodule-path>
hg bookmark master -r default
hg gexport # Can take quite a long time (~1 hour for carpet-hg)
hg pull git # Pull from the git submodule
hg push # Push to the upstream hg repo

This obviously has to be improved.

==Viewing history==

Each sub-repository is a separate fully-fledged Git repository, so you can go into each one and type "git status", "git log" etc. This works for repositories which are SVN upstream, because we use git-svn to convert the repositories to Git on the server. You can also use the usual graphical tools (we recommend GitX from [http://brotherbard.com/blog/2010/03/experimental-gitx-fork/] for Mac OS, and XXX for Linux) on each subrepository to visualise the log messages and patches, see which files have local changes, and interactively commit parts of files.

==Viewing local modifications==

You can see at a glance which of the submodules have local changes using

git status

You can then go into each submodule and see those changes using git status or git diff.

GitSuperRepoRationale

2011-06-27T08:35:17Z

Barry.wardell:

[[Category:GitSuperRepo]]

=Rationale=

''This section describes the motivations for this project and how our solution addresses problems with the existing systems.''

==Background==

* Einstein toolkit built from many different components living in their own repositories

* End user must check out each component and compile them together into an executable which is then run to produce output

* End user is often also a developer of some of the components (public or private)

* GetComponents (URL) is a tool to simplify this process by collecting component repository information into a single "CRL" file (CRL = Component Retrieval Language).

* GetComponents allows you to check out the latest versions from a CRL file, or to update an existing set of checkouts to the latest version

==Problems==

* Upstream projects use different version control systems (SVN, Git, Mercurial, ...) leading to a nonuniform experience for the end user/developer. Multiple tools must be learned for merging/branching/committing etc.

* It is not easy to see at a glance exactly what version of the code is in use. One could iterate over all the different repositories, of different types, and print the revision information, and any local differences. This could be added to GetComponents, but this has not been done yet and we argue that this is not the best solution to the problem.

* Knowing what version of the code has been used to produce a given scientific result is essential for the scientific process, where results must be repeatable. The current best solution to this problem is the Formaline thorn which stores a complete copy of the source code of all thorns in the simulation output directory. We argue that this is only a partial solution to the problem. While all the source code is present, the version control metadata has been entirely stripped. When comparing different simulations, at best one obtains a large diff of all the source changes between them, without information about why they were made or who made them. There is also no method for conveniently using the formaline output for a new simulation.

* Updating a Cactus source tree is currently an irreversible and dangerous process. There is no guarantee that the "current" trunk branch of all the components will function correctly, and there is no way, short of a manual backup beforehand, of reverting to the previous state if they don't. It is not possible to see, at a glance, exactly what will be updated when you run "GetComponents -u".

* It is desirable for different members of a scientific research group to be using the same version of the code for production simulations, or at least for this to be possible/easy. In the current setup, each user is responsible for managing their own Cactus tree and will likely have completely different versions of the code, depending on when they last updated. It is not even guaranteed that the code can be described by a single "checkout date", since different components could be checked out at different times. Users may also have applied patches or altered behaviour, fixing bugs or adding features, to any of the components.

* SVN does not allow distributed version control. Many components of the ET are in SVN, which means that users cannot use version control locally. They cannot commit locally frequently and go back to previous versions when there are problems, then bundle up the changes as a coherent commit to upstream.

* Managing branches is difficult. Suppose a user wants to implement a new feature. First, a new branch is created in the corresponding repository (we ignore here the fact that most of the components are in SVN, which does not encourage this mode of development), then the feature is committed bit by bit to that branch. In the course of development, the user might want to run a production simulation. So they would switch that repository back to the production branch temporarily. However, there is no global record of which branch each repository is currently on, and it might be that some repositories are not on "production" branches. The best solution at the moment is to have a separate production tree, but users rarely have the discipline to do this.

==Proposed solution==

We propose a solution based on the idea of a "Git Super-Repository" with each of the components (thorn repositories) linked into it as submodules (repository pointers).

* A single Git repository represents the complete state of the code

* Each group can set up their own super-repo on their server containing pointers to both the public repositories and their own private ones.

* Any non-git repositories are mirrored on the group's server (using git-svn, for example) as Git repositories so that users can interact with a single version control system.

* Standard git tools can be used to determine the current revision of the code and any local changes for storing in the simulation output directory for the purpose of reproducibility.

* It is easy to see what branch each repository is on, and to use branches with components which are hosted in SVN

* We ensure that committing back to the upstream repositories is easy

* A "tested" branch can be created in the super-repo. This branch will only be advanced when the corresponding revision has passed the automated build and test process. This means that a user can know beforehand that if they use this branch, they will always get a working version of the code.

* Updating is now safe and reversible. Since the current state is encapsulated in a revision ID, one can easily revert back to it after an update if the code no longer works.

* A group can have a "production" branch in the super-repo (possibly one per project). Switching to this branch switches all components at the same time, and it can be easily verified that the code is now all on the production branch. The production branch can be advanced in a controlled way when it has been tested, and all users can update to the new tested version.

* Comparing the codes used to produce two simulations is now easier. Assuming that the code revision information is present in the output directory, users can simply do a "git diff <rev1> <rev2>" between the two revisions. They can use standard graphical tools to see the log messages and authors of the commits, or look in full detail at the patches themselves. The question "did you run with XX fix?" can be trivially answered. The use of a production branch would also help in this.

* Users have all the benefits of distributed revision control, and the entire version history of all components (including SVN ones) at their fingertips in their local repository copies.

==Scenarios==

We now describe several scenarios where a standard Cactus checkout provides a poor user experience that is improved by using a Git super-repo.

===Updating a source tree===

A user has a standard Cactus checkout, for example of the Einstein Toolkit. They last updated it a few months ago, and would like to update it to be "current". The current method for doing this is to run GetComponents --update with the corresponding thornlist (assuming that the user has kept it). This will update all the repositories in the tree to the current latest version. There is no way to see what will be updated (e.g. commit messages of the changes) and no way to revert to the previous state after the update has occurred. There is no easy way to know whether the latest version of the code passes its test suites.

With a Git super-repository, a user can see the changes that would be obtained on an update:

git fetch
git submodule fetch ''I made this up - how would you really do it?''
git log --oneline HEAD..FETCH_HEAD

578f986 Fix indexing in 7th order prolongation
b8ab9dc fix testsuites
7f10f4b handle the case of only one point in a dimention in the code for the diagonal as well, in a similar way as for the other directi
3fc7a5b LoopControl: provide Fortran interface to vectorized loops
4f1cbfb CarpetLib: remove superfluous OMP PARALLEL section in (W)ENO prolongation
10a7e54 Revert "remove superfluous OMP PARALLEL section in (W)ENO prolongation"

''Is the HEAD..FETCH_HEAD the simplest way to do this? How does this work with submodules?''

The user can then do the update,

git pull
git submodule update

try out the code, and if it doesn't work, or they decide for some other reason that they want to go back to the old version, they can find that version from

git reflog

and revert to it:

git revert --hard <commit>

''Does this work with submodules? --hard is probably wrong - what is the best way to do this?''

In the fetch and pull commands above, the user could specify the "tested" branch which always points to the most recent revision that passes all the testsuites. This way, the user knows that the version they are getting will compile and pass all tests.

=== Source consistency in a project ===

Several people are collaborating on a project. With a standard checkout, there is currently no convenient way to ensure that all the people have the same version of the code when they run simulations for the project. One of the project members does some development work to try to get something to work, and wants to share that with the other members of the project (but not to commit it upstream, as it is still experimental). Currently this must be handled with patch files. Very soon, even if all the project members started off with the same source tree, they will likely have different versions due to local modifications, and no easy way to see what those changes are.

With a Git super-repo, a branch can be easily created in the group's local repository for the project (one global branch, not one per sub-repository). Each user can then use

git log --oneline projectbranch..

to see what commits they have locally that are not on the project branch, and

git diff

to see their local uncommitted changes across all repositories. Once the project members agree that certain features are required for the project, these can be pushed to the project branch and all the users can update. At all times, it is easy to see the difference between each user's trees.

''How much of this actually works with submodules?''

=== Identifying whether a fix has been applied for a given simulation ===

One member of a group has a problem with their simulation. It is crashing and they don't know why. They ask around, and someone thinks they may know the cause, and ask what version of the code they are using, specifically whether they have updated a certain thorn to include a particular fix. Unfortunately, the simulation had been sitting in the queue on a supercomputer for quite a while, and the user has updated their Cactus tree, so they don't know what version of the code was used to produce the simulation. With a standard checkout, the user must first find the corresponding fix (e.g. using svn log and svn diff), then untar the correspinding Formaline tarball from the simulation output directory, and see by eye if it looks like the fix has been applied. If there have been subsequent changes to the file, it might look completely different to when it was fixed, and it might not be easy to tell if the version of the code is from before or after the fix.

With a Git super-repo, the user can look in the simulation output directory where Formaline has stored the Git hash of the version of the code that was used, along with any local changes as patch files. They can then find the hash of the commit, and look for both in the "git log" output. If their version comes after the fix, then they know that they were running with the fix already.

=== Determining the cause of differences in a simulation ===

A scientist has a problem. They have run a new simulation and are getting results they don't understand. To track this down, they try to run an old simulation that worked, and find that they get different results. They now need to determine what has changed in the code between the two simulations. With a standard Cactus checkout, their only option is to untar all the Formaline tarballs and run a "diff" between the two. They will see all the source code changes in raw form. They will not know who has made the changes, or why. There is no realistic way to unapply a particular suspect commit and test the new version.

With a Git super-repo, the user will look in the simulation output directory and identify the Git hash of each version of the code. They can then run

git log --oneline <rev1>..<rev2>

to see all the commits which are different between the two versions. This will show the first line of the log message. The user can then select one which looks promising and view the patch in detail:

git log -p <rev>

This will show the full text of the diff, as well as the full description provided by the author, who is also identified. Perhaps after some further explanations by the patch author, the user could then revert this single patch

git revert <rev>

(this creates a new local commit which reverts the old patch) and try the simulation again, and determine if this patch was responsible for the problem or not.

''How much of this actually works with submodules?''

GitSuperRepo

2011-06-27T08:33:42Z

Barry.wardell:

[[Category:GitSuperRepo]]

The current method for checking out and updating components of the Einstein Toolkit has several shortcomings. We propose an additional layer of management around a Cactus tree achieved by storing it in a Git repository. This provides a uniform version control interface to the code with all the version control information from the source repositories available. It also allows the entire Cactus tree to be treated as a single repository, and allows "versions" of the code to be identified by a single Git revision.

This project is still in the early stages. We (Barry Wardell and Ian Hinder) are developing a set of techniques and tools which enable what we consider to be better workflows for using Cactus. As we use these ourselves, we will learn about what works and what doesn't, and hopefully will be able to provide recommended workflows for end-users.

==Documents==

* [[GitSuperRepoRationale|Rationale]]
* [[GitSuperRepoUsersGuide|User's guide]]
* [[GitSuperRepoServerGuide|Server administrator's guide]]
* [[GitSuperRepoUnsorted|Unsorted information]]

==Progress==

* We have set up a repository which implements most of what is described in the [[GitSuperRepoRationale|Rationale]].

* Repository is updated from the source repositories every hour

==Planned work==

* Figure out some good scripts or git aliases for making working with the submodules more straightforward.

* Set up a public "einsteintoolkit" repository

* Integrate with build-and-test system so that there is always a branch which passes build-and-test

* Implement a technique for storing version control information (revision plus differences) into simulation output

* Update the group repository based on commit emails or an RSS feed rather than hourly

GitSuperRepoUsersGuide

2011-06-24T05:35:25Z

Barry.wardell: /* Viewing local modifications */

''Include here simple user-friendly instructions for using a Cactus super-repository''

We provide a Git super-repo of the Einstein Toolkit. This page describes how to check it out and work with it. It is currently more of a tutorial than a User Guide.

Note that these instructions are a work in progress and will not currently work.

Much of this is too complicated to remember and understand. We need to make it much simpler, possibly by providing scripts.

==Checking out==

git clone --recursive cactus@git.einsteintoolkit.org:einsteintoolkit Cactus

This takes about 10 minutes and will create a directory called Cactus containing the entire Einstein Toolkit and all its version history.

==Updating==

To update the checkout, it is first a good idea to see what would be changed. You can get a list of the changes by

...

If this is OK, you can then update using:

cd Cactus
git pull
git submodule update

==Checking out a specific branch==

If you want to use a particular Einstein Toolkit release, perhaps because the current development version is unstable, use

git checkout ET_2011_06

The tree will be very quickly updated to match the release. All changed files will have updated datestamps, so you should be able to trust the Cactus make system to recompile only what is necessary. However, it might be safer to delete any configurations and build them again.

==Committing==

===Git===

For thorns whose source repositories are in Git already, you can commit and push as normal in the subrepositories. If they have been cloned from a public URL, as for the Einstein Toolkit URL above, you can use

cd <submodule-path>
git remote add upstream <new-url>
git push upstream ...

===SVN===

For subrepositories which are SVN upstream (for example simfactory), you must first initialise the subrepository for committing:

cd simfactory
git checkout trunk
git svn init -s --prefix=origin/ svn://...

You can now commit directly to the source SVN repository using

git svn dcommit ...

assuming you have commit rights in the source SVN repository.

===Mercurial===

For Mercurial upstream repositories, it is more complicated, as there is no git-hg (equivalent to git-svn):

hg clone <hg-url> <submodule-path>-upstream
cd <submodule-path>-upstream
vi .hg/hgrc # Add:
# [git]
# intree = 1
# [path]
# git = ../<submodule-path>
hg bookmark master -r default
hg gexport # Can take quite a long time (~1 hour for carpet-hg)
hg pull git # Pull from the git submodule
hg push # Push to the upstream hg repo

This obviously has to be improved.

==Viewing history==

Each sub-repository is a separate fully-fledged Git repository, so you can go into each one and type "git status", "git log" etc. This works for repositories which are SVN upstream, because we use git-svn to convert the repositories to Git on the server. You can also use the usual graphical tools (we recommend GitX from [http://brotherbard.com/blog/2010/03/experimental-gitx-fork/] for Mac OS, and XXX for Linux) on each subrepository to visualise the log messages and patches, see which files have local changes, and interactively commit parts of files.

==Viewing local modifications==

You can see at a glance which of the submodules have local changes using

git status

You can then go into each submodule and see those changes using git status or git diff.

GitSuperRepoUsersGuide

2011-06-24T05:27:57Z

Barry.wardell: /* Viewing local modifications */

''Include here simple user-friendly instructions for using a Cactus super-repository''

We provide a Git super-repo of the Einstein Toolkit. This page describes how to check it out and work with it. It is currently more of a tutorial than a User Guide.

Note that these instructions are a work in progress and will not currently work.

Much of this is too complicated to remember and understand. We need to make it much simpler, possibly by providing scripts.

==Checking out==

git clone --recursive cactus@git.einsteintoolkit.org:einsteintoolkit Cactus

This takes about 10 minutes and will create a directory called Cactus containing the entire Einstein Toolkit and all its version history.

==Updating==

To update the checkout, it is first a good idea to see what would be changed. You can get a list of the changes by

...

If this is OK, you can then update using:

cd Cactus
git pull
git submodule update

==Checking out a specific branch==

If you want to use a particular Einstein Toolkit release, perhaps because the current development version is unstable, use

git checkout ET_2011_06

The tree will be very quickly updated to match the release. All changed files will have updated datestamps, so you should be able to trust the Cactus make system to recompile only what is necessary. However, it might be safer to delete any configurations and build them again.

==Committing==

===Git===

For thorns whose source repositories are in Git already, you can commit and push as normal in the subrepositories. If they have been cloned from a public URL, as for the Einstein Toolkit URL above, you can use

cd <submodule-path>
git remote add upstream <new-url>
git push upstream ...

===SVN===

For subrepositories which are SVN upstream (for example simfactory), you must first initialise the subrepository for committing:

cd simfactory
git checkout trunk
git svn init -s --prefix=origin/ svn://...

You can now commit directly to the source SVN repository using

git svn dcommit ...

assuming you have commit rights in the source SVN repository.

===Mercurial===

For Mercurial upstream repositories, it is more complicated, as there is no git-hg (equivalent to git-svn):

hg clone <hg-url> <submodule-path>-upstream
cd <submodule-path>-upstream
vi .hg/hgrc # Add:
# [git]
# intree = 1
# [path]
# git = ../<submodule-path>
hg bookmark master -r default
hg gexport # Can take quite a long time (~1 hour for carpet-hg)
hg pull git # Pull from the git submodule
hg push # Push to the upstream hg repo

This obviously has to be improved.

==Viewing history==

Each sub-repository is a separate fully-fledged Git repository, so you can go into each one and type "git status", "git log" etc. This works for repositories which are SVN upstream, because we use git-svn to convert the repositories to Git on the server. You can also use the usual graphical tools (we recommend GitX from [http://brotherbard.com/blog/2010/03/experimental-gitx-fork/] for Mac OS, and XXX for Linux) on each subrepository to visualise the log messages and patches, see which files have local changes, and interactively commit parts of files.

==Viewing local modifications==

You can see at a glance which of the submodules have local uncommitted changes using

git status

You can then go into each submodule and see those changes using git status or git diff.