Test suite results are unwieldy - Revision history

Knarf at 13:19, 4 November 2011

2011-11-04T13:19:50Z

Knarf at 13:15, 4 November 2011

2011-11-04T13:15:54Z

Bmundim at 11:37, 14 October 2011

2011-10-14T11:37:17Z

Bmundim at 11:36, 14 October 2011

2011-10-14T11:36:53Z

Eschnett at 10:13, 14 October 2011

2011-10-14T10:13:10Z

Eschnett at 10:12, 14 October 2011

2011-10-14T10:12:44Z

New page

They consume disk space, they consume time when downloading and when copying to a remote machine. People view them as "unimportant" (when compared to source files), and are generally annoyed. What can we do about this?

Some suggestions include:
* Do nothing; explain to people that test suite results are very important to ensure correctness, tell them to suffer through transmit times and ask for larger quota
* * That's not a solution.
* Con't copy test suite results to remote machines by default; only copy them when requested
* * This makes it more difficult to run the test suite, which should (ideally) be run every time a new executable has been created for a production run.
* tar/gzip test suite results to save disk space, possibly also storing them this way in the SVN repository
* * This doesn't save transmit time (since rsync already compresses), and (if used in the repository) doesn't allow delta compression
* Combine test suite results from different source trees into a single directory (with soft links or hard links or similarly)
* * This would be an extra step that runs after all files are there, already consuming disk space.
* Use HDF5 instead of ASCII
* * Binary files are not well handled by SVN.
* Combine several ASCII files into one (fewer files are handled more efficiently), e.g. designing a simple ASCII container format for test cases, or even using this format in CarpetIOASCII
* * Wouldn't save disk space.
* Use a different ASCII format that doesn't store 13 columns for 1 data field
* * Maybe we could use the ygraph format?

←Older revision		Revision as of 13:19, 4 November 2011
Line 21:		Line 21:
	** The test suite data could be laid out similarly to the source tree: ./Cactus/arrangements/ARRANGEMENTNAME/THORNNAME ==> ./Cactus/testsuites/ARRANGEMENTNAME/THORNNAME		** The test suite data could be laid out similarly to the source tree: ./Cactus/arrangements/ARRANGEMENTNAME/THORNNAME ==> ./Cactus/testsuites/ARRANGEMENTNAME/THORNNAME
	** One problem with that is that there is no clear connection between a version of the source and a version of a testsuite anymore.		** One problem with that is that there is no clear connection between a version of the source and a version of a testsuite anymore.
		+	* Don't change the layout of a usual checkout, but let simfactory not only omit .svn directories, but let it also compress the testsuites to single compressed files.
		+	** This would not significantly reduce the time for syncing - in fact it might add a bit because it would need to compress them locally in the first place.
		+	** It would, on the remote side, be no problem for version control, since there is none anyway (.svn is not copied)
		+	** It should reduce IO-stress on machines like kraken/ranger where the number of files is the problem, not so much their size.
		+	** Running the testsuites remotely would still work (since Cactus now supports that).

←Older revision		Revision as of 13:15, 4 November 2011
Line 20:		Line 20:
	** The user could choose not to checkout the data if she has a laptop with very little hard drive space left, for example.		** The user could choose not to checkout the data if she has a laptop with very little hard drive space left, for example.
	** The test suite data could be laid out similarly to the source tree: ./Cactus/arrangements/ARRANGEMENTNAME/THORNNAME ==> ./Cactus/testsuites/ARRANGEMENTNAME/THORNNAME		** The test suite data could be laid out similarly to the source tree: ./Cactus/arrangements/ARRANGEMENTNAME/THORNNAME ==> ./Cactus/testsuites/ARRANGEMENTNAME/THORNNAME
		+	** One problem with that is that there is no clear connection between a version of the source and a version of a testsuite anymore.

@@ Line 4: / Line 4: @@
 * Do nothing; explain to people that test suite results are very important to ensure correctness, tell them to suffer through transmit times and ask for larger quota
 ** That's not a solution.
-* Con't copy test suite results to remote machines by default; only copy them when requested
+* Don't copy test suite results to remote machines by default; only copy them when requested
 ** This makes it more difficult to run the test suite, which should (ideally) be run every time a new executable has been created for a production run.
 * tar/gzip test suite results to save disk space, possibly also storing them this way in the SVN repository

←Older revision		Revision as of 11:36, 14 October 2011
Line 16:		Line 16:
	* Use a different ASCII format that doesn't store 13 columns for 1 data field		* Use a different ASCII format that doesn't store 13 columns for 1 data field
	** Maybe we could use the ygraph format?		** Maybe we could use the ygraph format?
		+	* BCM: Separate the test suite results (data) from the source tree.
		+	** This allows a unified management of the data: when to check it out, when to sync it to a remote machine, etc.
		+	** The user could choose not to checkout the data if she has a laptop with very little hard drive space left, for example.
		+	** The test suite data could be laid out similarly to the source tree: ./Cactus/arrangements/ARRANGEMENTNAME/THORNNAME ==> ./Cactus/testsuites/ARRANGEMENTNAME/THORNNAME

@@ Line 3: / Line 3: @@
 Some suggestions include:
 * Do nothing; explain to people that test suite results are very important to ensure correctness, tell them to suffer through transmit times and ask for larger quota
-* * That's not a solution.
+** That's not a solution.
 * Con't copy test suite results to remote machines by default; only copy them when requested
-* * This makes it more difficult to run the test suite, which should (ideally) be run every time a new executable has been created for a production run.
+** This makes it more difficult to run the test suite, which should (ideally) be run every time a new executable has been created for a production run.
 * tar/gzip test suite results to save disk space, possibly also storing them this way in the SVN repository
-* * This doesn't save transmit time (since rsync already compresses), and (if used in the repository) doesn't allow delta compression
+** This doesn't save transmit time (since rsync already compresses), and (if used in the repository) doesn't allow delta compression
 * Combine test suite results from different source trees into a single directory (with soft links or hard links or similarly)
-* * This would be an extra step that runs after all files are there, already consuming disk space.
+** This would be an extra step that runs after all files are there, already consuming disk space.
 * Use HDF5 instead of ASCII
-* * Binary files are not well handled by SVN.
+** Binary files are not well handled by SVN.
 * Combine several ASCII files into one (fewer files are handled more efficiently), e.g. designing a simple ASCII container format for test cases, or even using this format in CarpetIOASCII
-* * Wouldn't save disk space.
+** Wouldn't save disk space.
 * Use a different ASCII format that doesn't store 13 columns for 1 data field
-* * Maybe we could use the ygraph format?
+** Maybe we could use the ygraph format?