Improving the treatment of external libraries - Revision history

Bmundim: /* Additional Issues */

2015-10-21T14:30:23Z

‎Additional Issues

Rhaas: comment on not storing tarbal on our own servers

2015-06-10T00:08:44Z

comment on not storing tarbal on our own servers

Hinder at 14:24, 9 June 2015

2015-06-09T14:24:15Z

38.117.109.20: /* Possible improvements */

2012-11-13T17:21:39Z

‎Possible improvements

38.117.109.20: /* Questions */

2012-11-12T21:34:51Z

‎Questions

38.117.109.20 at 21:31, 12 November 2012

2012-11-12T21:31:19Z

Hinder: /* Factoring out common code */

2012-11-12T21:00:28Z

‎Factoring out common code

Hinder: /* Factoring out common code */

2012-11-12T20:59:16Z

‎Factoring out common code

Hinder: /* Proposal */

2012-11-12T20:58:04Z

‎Proposal

Hinder: /* Requirements */

2012-11-12T20:49:41Z

‎Requirements

@@ Line 67: / Line 67: @@
 to be worked into the text above.
-* "Standard" paths such as those with prefix /usr or /usr/local must not be added to INC_DIRS and LIB_DIRS.
+* "Standard" paths such as those with prefix /usr or /usr/local must not be added to INC_DIRS and LIB_DIRS (BCM: or at least it should be possible to optionally add them at the end)
 * Instead of a capitalised library name, we should pass the "natural" name, and capitalise within the script. This allows nicer error messages.
 * Assuming that configure_library unpacks the library: It will be necessary to pass in also the directory in which the (unpacked) configure script is found.

@@ Line 97: / Line 97: @@
 The current system of storing the source tarball in the thorn SVN repository is not ideal.  Included here is an email written by Ian in 2013 about this, with some ideas for improving the system.
-* The tarball would not be included in the repository
+* The tarball would not be included in the repository. RH: I would prefer to store tarballs (or their content) in a repository we control, this does not have to be the same that holds the control files, the reason is that some of the ExternalLibraries may change their hosting url/not life as long as the ET. For example the oldest version of curl still available on the webserver at http://curl.haxx.se/ is from 2000 and anything older than 2010 is in a subdirectory archeology.
 * The configuration.ccl file of the thorn would have a field for the URL of a tarball to download
 * At configuration time, the thorn's configure script (or something factored into a Cactus script) would download the tarball using the URL if it is necessary to build the library on that machine.
 * The tarball would be cached in Cactus/librarycache or similar
-* A script or makefile target would be provided to "pre-cache" the tarball of one or more thorns
+* A script or makefile target would be provided to "pre-cache" the tarball of one or more thorns. RH: GetComponents can download arbitrary single files via http so we already have that possibility
 A checkout of the ET now would not download any external library tarballs.  If, on a given machine, a library is not installed and needs to be built, it would be downloaded when needed. If you want to make sure you have everything needed to build, for example if you are about to catch a plane, you could run something like "make get-libraries" and all library tarballs would be downloaded into Cactus/librarycache.  [It would be possible to do this using simfactory, which might be able to determine via its machine database which machines need which tarballs downloaded].  You could then choose to sync this to the remote machine, or not.  On a remote machine, you could run get-libraries on the head node which presumably has an internet connection so that the library tarballs are available to the build process which might, as Frank points out, not have internet access.
 The vast majority of users would notice little difference; the checkout would be faster, and less disk space would be used by their Cactus trees.  Edge-case users who have to compile on machines without internet connections would have a simple command to run on the head node of the machine which would restore the same functionality that we currently have.

←Older revision		Revision as of 14:24, 9 June 2015
Line 90:		Line 90:

	* Do we really want to continue to develop this mechanism? Or can we adopt rpm / dpkg / pkgconfig instead? In the long run, it may be easier to write a pkgconfig definition for LORENE than maintaining whole sets of configuration scripts ourselves.		* Do we really want to continue to develop this mechanism? Or can we adopt rpm / dpkg / pkgconfig instead? In the long run, it may be easier to write a pkgconfig definition for LORENE than maintaining whole sets of configuration scripts ourselves.
		+
		+	== Building from source ==
		+
		+	(move this to a separate page)
		+
		+	The current system of storing the source tarball in the thorn SVN repository is not ideal. Included here is an email written by Ian in 2013 about this, with some ideas for improving the system.
		+
		+	* The tarball would not be included in the repository
		+	* The configuration.ccl file of the thorn would have a field for the URL of a tarball to download
		+	* At configuration time, the thorn's configure script (or something factored into a Cactus script) would download the tarball using the URL if it is necessary to build the library on that machine.
		+	* The tarball would be cached in Cactus/librarycache or similar
		+	* A script or makefile target would be provided to "pre-cache" the tarball of one or more thorns
		+
		+	A checkout of the ET now would not download any external library tarballs. If, on a given machine, a library is not installed and needs to be built, it would be downloaded when needed. If you want to make sure you have everything needed to build, for example if you are about to catch a plane, you could run something like "make get-libraries" and all library tarballs would be downloaded into Cactus/librarycache. [It would be possible to do this using simfactory, which might be able to determine via its machine database which machines need which tarballs downloaded]. You could then choose to sync this to the remote machine, or not. On a remote machine, you could run get-libraries on the head node which presumably has an internet connection so that the library tarballs are available to the build process which might, as Frank points out, not have internet access.
		+
		+	The vast majority of users would notice little difference; the checkout would be faster, and less disk space would be used by their Cactus trees. Edge-case users who have to compile on machines without internet connections would have a simple command to run on the head node of the machine which would restore the same functionality that we currently have.

←Older revision		Revision as of 17:21, 13 November 2012
Line 82:		Line 82:

	* In most cases, a library that has been built once can be re-used by all other builds on the same machine. (Exceptions exist.) Since reducing build time is mentioned as a goal above, we could install libraries into a specific directory (e.g. Cactus/exe/lib), and thus share external library builds between configurations. These would then not be deleted by a make realclean, but would be rebuilt when necessary. It is up to the option list to ensure that incompatible option lists specify different install directories. Maybe the option list name itself could be used as prefix.		* In most cases, a library that has been built once can be re-used by all other builds on the same machine. (Exceptions exist.) Since reducing build time is mentioned as a goal above, we could install libraries into a specific directory (e.g. Cactus/exe/lib), and thus share external library builds between configurations. These would then not be deleted by a make realclean, but would be rebuilt when necessary. It is up to the option list to ensure that incompatible option lists specify different install directories. Maybe the option list name itself could be used as prefix.
		+	* We should separate the distributed tarballs from the thorns containing the logic. This would save download time for those who don't want / need the tarballs.
		+	* We could allow the tarballs to be downloaded directly and only when needed, without storing them in svn.

	===Some high-level thoughts===		===Some high-level thoughts===

@@ Line 65: / Line 65: @@
 ===Additional Issues===
-To be worked into the text above.
+to be worked into the text above.
-* "Standard" paths such as those with prefix /usr or /usr/local should not be added to INC_DIRS and LIB_DIRS.
+* "Standard" paths such as those with prefix /usr or /usr/local must not be added to INC_DIRS and LIB_DIRS.
 * Instead of a capitalised library name, we should pass the "natural" name, and capitalise within the script. This allows nicer error messages.
-* Assuming that configure_library unpacks the library: It may be necessary to pass in also the directory in which the (unpacked) configure script is found.
+* Assuming that configure_library unpacks the library: It will be necessary to pass in also the directory in which the (unpacked) configure script is found.
 * <patch>: Note that multiple patches may be required.
 * Running a test program: Sometimes it is not possible to run a program (e.g. for MPI). In this case, building and linking is still a good idea for testing.
-* We should think about cross-compiling. It needs to be supported at least to the extent that the configuration options specify an existing library, which is then accepted by Cactus.
+* We should think about cross-compiling. It needs to be supported at least to the extent that the configuration options can specify an existing library, which is then accepted by Cactus.
 * Maybe there should be a generic mechanism to skip all testing?
-* Having a _DIR variable contains "BUILD" and "NO_BUILD" doesn't really make sense. Maybe we should set e.g. "HDF5=BUILD", or "HDF5=NO_BUILD", and use HDF5_DIR only for providing default values for HDF5_LIB_DIRS? In this case, "HDF5=NO_TEST" would also be possible.
+* Having a _DIR variable containing "BUILD" and "NO_BUILD" doesn't really make sense. Maybe we should set e.g. "HDF5=BUILD", or "HDF5=NO_BUILD", and use HDF5_DIR only for providing default values for HDF5_LIB_DIRS? In this case, "HDF5=NO_TEST" would also be possible.
 * Certain environment variables (e.g. LIBS) should automatically be unset by configure_library, not by the thorn's build script. These variables have a special, non-standard meaning in Cactus, and basically all configure scripts get confused by them.
-===Possible improvements====
+===Possible improvements===
 which are not really related to the topic discussed here.
-* In most cases, a library that has been built once can be re-used by all other builds on the same machine. (Exceptions exist.) Since reducing build time is mentioned as a goal above, we could install libraries into a specific directory (e.g. Cactus/exe/lib), and thus share external library builds between configurations. These would then not be deleted by a make realclean, but would be rebuilt when necessary. It is up to the option list to ensure that incompatible option lists to specify different install directories.
+* In most cases, a library that has been built once can be re-used by all other builds on the same machine. (Exceptions exist.) Since reducing build time is mentioned as a goal above, we could install libraries into a specific directory (e.g. Cactus/exe/lib), and thus share external library builds between configurations. These would then not be deleted by a make realclean, but would be rebuilt when necessary. It is up to the option list to ensure that incompatible option lists specify different install directories. Maybe the option list name itself could be used as prefix.
 ===Some high-level thoughts===
@@ Line 87: / Line 87: @@
 which should probably be at the top.
-* Do we really want to continue to develop this mechanism? Or can we adopt rpm / dpkg / pkgconfig instead? In the long run, it may be easier to write a pkgconfig definition for LORENE than maintaining whole configuration scripts ourselves.
+* Do we really want to continue to develop this mechanism? Or can we adopt rpm / dpkg / pkgconfig instead? In the long run, it may be easier to write a pkgconfig definition for LORENE than maintaining whole sets of configuration scripts ourselves.

←Older revision		Revision as of 21:31, 12 November 2012
Line 62:		Line 62:

	* How should the user get the behaviour of setting <lib>_DIR to point to a directory, i.e. explicitly using a specific installation, if there is no such directory, and the include and lib dirs need to be specified separately? Maybe setting any of these three variables (<lib>_DIR, <lib>_INC_DIRS and <lib>_LIB_DIRS) should trigger using exactly that version of the library.		* How should the user get the behaviour of setting <lib>_DIR to point to a directory, i.e. explicitly using a specific installation, if there is no such directory, and the include and lib dirs need to be specified separately? Maybe setting any of these three variables (<lib>_DIR, <lib>_INC_DIRS and <lib>_LIB_DIRS) should trigger using exactly that version of the library.
		+
		+	===Additional Issues===
		+
		+	To be worked into the text above.
		+
		+	* "Standard" paths such as those with prefix /usr or /usr/local should not be added to INC_DIRS and LIB_DIRS.
		+	* Instead of a capitalised library name, we should pass the "natural" name, and capitalise within the script. This allows nicer error messages.
		+	* Assuming that configure_library unpacks the library: It may be necessary to pass in also the directory in which the (unpacked) configure script is found.
		+	* <patch>: Note that multiple patches may be required.
		+	* Running a test program: Sometimes it is not possible to run a program (e.g. for MPI). In this case, building and linking is still a good idea for testing.
		+	* We should think about cross-compiling. It needs to be supported at least to the extent that the configuration options specify an existing library, which is then accepted by Cactus.
		+	* Maybe there should be a generic mechanism to skip all testing?
		+	* Having a _DIR variable contains "BUILD" and "NO_BUILD" doesn't really make sense. Maybe we should set e.g. "HDF5=BUILD", or "HDF5=NO_BUILD", and use HDF5_DIR only for providing default values for HDF5_LIB_DIRS? In this case, "HDF5=NO_TEST" would also be possible.
		+	* Certain environment variables (e.g. LIBS) should automatically be unset by configure_library, not by the thorn's build script. These variables have a special, non-standard meaning in Cactus, and basically all configure scripts get confused by them.
		+
		+	===Possible improvements====
		+
		+	which are not really related to the topic discussed here.
		+
		+	* In most cases, a library that has been built once can be re-used by all other builds on the same machine. (Exceptions exist.) Since reducing build time is mentioned as a goal above, we could install libraries into a specific directory (e.g. Cactus/exe/lib), and thus share external library builds between configurations. These would then not be deleted by a make realclean, but would be rebuilt when necessary. It is up to the option list to ensure that incompatible option lists to specify different install directories.
		+
		+	===Some high-level thoughts===
		+
		+	which should probably be at the top.
		+
		+	* Do we really want to continue to develop this mechanism? Or can we adopt rpm / dpkg / pkgconfig instead? In the long run, it may be easier to write a pkgconfig definition for LORENE than maintaining whole configuration scripts ourselves.

@@ Line 57: / Line 57: @@
 ** Search for the library, and use it if found, otherwise abort
 ** Search for the library, use if found, otherwise build
-* Set Cactus include and link variables so that the library will be appropriately found by Cactus
+* Set Cactus include and link variables so that the library will be appropriately found by Cactus, as well as the HAVE_<lib> macro
 ==Questions==
 * How should the user get the behaviour of setting <lib>_DIR to point to a directory, i.e. explicitly using a specific installation, if there is no such directory, and the include and lib dirs need to be specified separately?  Maybe setting any of these three variables (<lib>_DIR, <lib>_INC_DIRS and <lib>_LIB_DIRS) should trigger using exactly that version of the library.

@@ Line 42: / Line 42: @@
 ===Factoring out common code===
-Each library thorn's configure.sh script can call a bash function (or maybe a script) which we provide in the flesh: configure_library.  This function has the following semantics:
+Each library thorn's configure.sh script can call a bash function (or maybe a script) which we provide in the flesh: configure_library:
    configure_library <lib> <dist> <patch>...
-  <lib>: A capitalised name of the library, e.g. HDF5, HWLOC, MPI suitable for use in environment variable names.
+* <lib>: A capitalised name of the library, e.g. HDF5, HWLOC, MPI suitable for use in environment variable names.
-  <dist>: A path to the source distribution tarball relative to the thorn, e.g. dist/zlib-1.2.7.tar.gz
+* <dist>: A path to the source distribution tarball relative to the thorn, e.g. dist/zlib-1.2.7.tar.gz
-  <patch>: A path to any patch that needs to be applied
+* <patch>: A path to any patch that needs to be applied
-  configure_library does the following:
+configure_library does the following:
-  * Enable verbose output if requested
+* Enable verbose output if requested
-  * Look at the variables set by the user and choose which of the following to do, and do it:
+* Look at the variables set by the user and choose which of the following to do, and do it:
-  ** Build the library
+** Build the library
-  ** Use the library from a specified location, and test that this works
+** Use the library from a specified location, and test that this works, otherwise abort
-  ** Search for the library, use if found, otherwise build
+** Search for the library, and use it if found, otherwise abort
-  * Set Cactus include and link variables so that the library will be appropriately found by Cactus
+** Search for the library, use if found, otherwise build
 ==Questions==
 * How should the user get the behaviour of setting <lib>_DIR to point to a directory, i.e. explicitly using a specific installation, if there is no such directory, and the include and lib dirs need to be specified separately?  Maybe setting any of these three variables (<lib>_DIR, <lib>_INC_DIRS and <lib>_LIB_DIRS) should trigger using exactly that version of the library.

@@ Line 21: / Line 21: @@
 ** is unset, or is the empty string, in which case the library is first searched for, and if not found, is built (we could have a synonym AUTO (caseless) for this, to make it explicit)
 * Provide variables which point to the include and library directories, but by default set these variables to <lib>/include and <lib>/lib.
 ==Questions==
 * How should the user get the behaviour of setting <lib>_DIR to point to a directory, i.e. explicitly using a specific installation, if there is no such directory, and the include and lib dirs need to be specified separately?  Maybe setting any of these three variables (<lib>_DIR, <lib>_INC_DIRS and <lib>_LIB_DIRS) should trigger using exactly that version of the library.