Organizing Tests into Suites - Revision history

Ivailop: /* Aggregating Report Output Across Multiple Stride Invocations */

2011-11-24T23:06:17Z

Aggregating Report Output Across Multiple Stride Invocations

← Older revision		Revision as of 23:06, 24 November 2011
Line 134:		Line 134:

	<pre>		<pre>
	> stride -O runComponentA.opt --upload=start		> stride --options_file runComponentA.opt --upload=start
	> stride -O runComponentB.opt --upload=add		> stride --options_file runComponentB.opt --upload=add
	> ..		> ..
	> stride -O runComponentN.opt --upload=finish		> stride --options_file runComponentN.opt --upload=finish

	</pre>		</pre>

Ivailop: /* Putting It All Together */

2011-11-24T23:05:48Z

Putting It All Together

← Older revision		Revision as of 23:05, 24 November 2011
Line 123:		Line 123:
	<source lang="dos">		<source lang="dos">
	stride --list --database TestApp.sidb \| perl OrganizeIntoSuites.pl > TestsInSuites.txt		stride --list --database TestApp.sidb \| perl OrganizeIntoSuites.pl > TestsInSuites.txt
	stride --database TestApp.sidb --device TCP:localhost:8000 -O TestsInSuites.txt		stride --database TestApp.sidb --device TCP:localhost:8000 --options_file TestsInSuites.txt
	</source>		</source>

Marku: /* Automating Your Test Unit Organization */

2011-08-29T16:41:12Z

Automating Your Test Unit Organization

← Older revision		Revision as of 16:41, 29 August 2011
Line 35:		Line 35:
	The naming convention follows the suite specification pattern: ''Name segments describing a hierarchy are delimited by a unique character or character sequence''.		The naming convention follows the suite specification pattern: ''Name segments describing a hierarchy are delimited by a unique character or character sequence''.

	For C++ environments where namespaces are used, the scope operator (<tt>::</tt>) makes a convenient delimiter. For other environments, the underscore (<tt>_</tt>) is a good choice. The [[~~Test Unit Samples~~\|Sample Tests]] use these naming conventions and can provide a convenient testbed to see how the automation technique works.		For C++ environments where namespaces are used, the scope operator (<tt>::</tt>) makes a convenient delimiter. For other environments, the underscore (<tt>_</tt>) is a good choice. The [[C/C%2B%2B_Samples\|Sample Tests]] use these naming conventions and can provide a convenient testbed to see how the automation technique works.

	Taking a couple of example test units from the [[~~Test Unit Samples~~\|Sample Tests]]:		Taking a couple of example test units from the [[C/C%2B%2B_Samples\|Sample Tests]]:

	s2_testclass::RuntimeServices::Simple --translates to--> -r /s2/testclass/RuntimeServices{s2_testclass::RuntimeServices::Simple}		s2_testclass::RuntimeServices::Simple --translates to--> -r /s2/testclass/RuntimeServices{s2_testclass::RuntimeServices::Simple}

Ivailop: /* Overview */

2010-05-17T21:24:03Z

Overview

← Older revision		Revision as of 21:24, 17 May 2010
Line 1:		Line 1:
	==Overview==		==Overview==
	Unless you specify otherwise, when [[Running Tests\|running tests]] stride will publish results from all of your results into a single top-level test suite. This is a satisfactory arrangement as long as the number of tests is relatively small. As the number of test units grows, or test units from disparate groups are aggregated (as in a Continuous Integration arrangement) this flat report organization becomes hard to use.		Unless you specify otherwise, when [[Running Tests\|running tests]], stride will publish results from all of your results into a single top-level test suite. This is a satisfactory arrangement as long as the number of tests is relatively small. As the number of test units grows, or test units from disparate groups are aggregated (as in a Continuous Integration arrangement) this flat report organization becomes hard to use.

	The software under test itself is typically broken up into separate units (e.g. libraries, components, objects, etc.) and organized into a hierarchy that reflects the functional organization of the software. If the results from STRIDE tests could be organized identically to the software under test, understanding and analyzing test results would be much simpler.		The software under test itself is typically broken up into separate units (e.g. libraries, components, objects, etc.) and organized into a hierarchy that reflects the functional organization of the software. If the results from STRIDE tests could be organized identically to the software under test, understanding and analyzing test results would be much simpler.

Ivailop: /* Overview */

2010-05-17T20:48:38Z

Overview

← Older revision		Revision as of 20:48, 17 May 2010
Line 1:		Line 1:
	==Overview==		==Overview==
	Unless you specify otherwise, when [[Running Tests\|running tests]] stride will publish results from all of your ~~tests~~ into a single top-level test suite ~~when run~~. This is a satisfactory arrangement as long as the number of tests is relatively small. As the number of test units grows, or test units from disparate groups are aggregated (as in a Continuous Integration arrangement) this flat report organization becomes hard to use.		Unless you specify otherwise, when [[Running Tests\|running tests]] stride will publish results from all of your results into a single top-level test suite. This is a satisfactory arrangement as long as the number of tests is relatively small. As the number of test units grows, or test units from disparate groups are aggregated (as in a Continuous Integration arrangement) this flat report organization becomes hard to use.

	The software under test itself is typically broken up into separate units (e.g. libraries, components, objects, etc.) and organized into a hierarchy that reflects the functional organization of the software. If the results from STRIDE tests could be organized identically to the software under test, understanding and analyzing test results would be much simpler.		The software under test itself is typically broken up into separate units (e.g. libraries, components, objects, etc.) and organized into a hierarchy that reflects the functional organization of the software. If the results from STRIDE tests could be organized identically to the software under test, understanding and analyzing test results would be much simpler.

Ivailop: /* Overview */

2010-05-17T20:47:54Z

Overview

← Older revision		Revision as of 20:47, 17 May 2010
Line 1:		Line 1:
	==Overview==		==Overview==
	Unless you specify otherwise, when [[running tests]] stride will publish results from all of your tests into a single top-level test suite when run. This is a satisfactory arrangement as long as the number of tests is relatively small. As the number of test units grows, or test units from disparate groups are aggregated (as in a Continuous Integration arrangement) this flat report organization becomes hard to use.		Unless you specify otherwise, when [[Running Tests\|running tests]] stride will publish results from all of your tests into a single top-level test suite when run. This is a satisfactory arrangement as long as the number of tests is relatively small. As the number of test units grows, or test units from disparate groups are aggregated (as in a Continuous Integration arrangement) this flat report organization becomes hard to use.

	The software under test itself is typically broken up into separate units (e.g. libraries, components, objects, etc.) and organized into a hierarchy that reflects the functional organization of the software. If the results from STRIDE tests could be organized identically to the software under test, understanding and analyzing test results would be much simpler.		The software under test itself is typically broken up into separate units (e.g. libraries, components, objects, etc.) and organized into a hierarchy that reflects the functional organization of the software. If the results from STRIDE tests could be organized identically to the software under test, understanding and analyzing test results would be much simpler.

Ivailop: /* Overview */

2010-05-17T20:47:15Z

Overview

← Older revision		Revision as of 20:47, 17 May 2010
Line 1:		Line 1:
	==Overview==		==Overview==
	Unless you specify otherwise, stride will publish results from all of your tests into a single top-level test suite when run. This is a satisfactory arrangement as long as the number of tests is relatively small. As the number of test units grows, or test units from disparate groups are aggregated (as in a Continuous Integration arrangement) this flat report organization becomes hard to use.		Unless you specify otherwise, when [[running tests]] stride will publish results from all of your tests into a single top-level test suite when run. This is a satisfactory arrangement as long as the number of tests is relatively small. As the number of test units grows, or test units from disparate groups are aggregated (as in a Continuous Integration arrangement) this flat report organization becomes hard to use.

	The software under test itself is typically broken up into separate units (e.g. libraries, components, objects, etc.) and organized into a hierarchy that reflects the functional organization of the software. If the results from STRIDE tests could be organized identically to the software under test, understanding and analyzing test results would be much simpler.		The software under test itself is typically broken up into separate units (e.g. libraries, components, objects, etc.) and organized into a hierarchy that reflects the functional organization of the software. If the results from STRIDE tests could be organized identically to the software under test, understanding and analyzing test results would be much simpler.


	== Organizing Into Suites at Test Runtime ==		== Organizing Into Suites at Test Runtime ==

Ivailop: /* Organizing Into Suites at Test Runtime */

2010-05-17T20:35:58Z

Organizing Into Suites at Test Runtime

← Older revision		Revision as of 20:35, 17 May 2010
Line 6:		Line 6:

	== Organizing Into Suites at Test Runtime ==		== Organizing Into Suites at Test Runtime ==
	The stride test runner allows you to specify a "path" under which a tests results will be published. The path is a forward-slash delimited list of suite names. The path is optionally specified in the argument to the <tt>-r</tt> command line option.		The [[STRIDE Runner\|stride test runner]] allows you to specify a "path" under which a tests results will be published. The path is a forward-slash delimited list of suite names. The path is optionally specified in the argument to the <tt>-r</tt> command line option.

	Following are some examples. The examples shown apply to both host script test modules and target test units. Many more examples can be seen [[Stride_Runner#Input\|here]]:.		Following are some examples. The examples shown apply to both host script test modules and target test units. Many more examples can be seen [[Stride_Runner#Input\|here]]:.

Marku: /* Notes */

2010-04-29T03:15:27Z

Notes

← Older revision		Revision as of 03:15, 29 April 2010
Line 149:		Line 149:
	<references/>		<references/>

	[[Category:Running Tests ~~& Publishing Results~~]]		[[Category:Running Tests]]

Mikee at 21:54, 28 January 2010

2010-01-28T21:54:58Z

← Older revision		Revision as of 21:54, 28 January 2010
Line 1:		Line 1:
	==Overview==		==Overview==
	Unless you specify otherwise, stride will publish results from all of your ~~test units~~ into a single top-level test suite when run. This is a satisfactory arrangement as long as the number of tests is relatively small. As the number of test units grows, or test units from disparate groups are aggregated (as in a Continuous Integration arrangement) this flat report organization becomes hard to use.		Unless you specify otherwise, stride will publish results from all of your tests into a single top-level test suite when run. This is a satisfactory arrangement as long as the number of tests is relatively small. As the number of test units grows, or test units from disparate groups are aggregated (as in a Continuous Integration arrangement) this flat report organization becomes hard to use.

	The software under test itself is typically broken up into separate units (e.g. libraries, components, objects, etc.) and organized into a hierarchy that reflects the functional organization of the software. If the results from STRIDE tests could be organized identically to the software under test, understanding and analyzing test results would be much simpler.		The software under test itself is typically broken up into separate units (e.g. libraries, components, objects, etc.) and organized into a hierarchy that reflects the functional organization of the software. If the results from STRIDE tests could be organized identically to the software under test, understanding and analyzing test results would be much simpler.
Line 6:		Line 6:

	== Organizing Into Suites at Test Runtime ==		== Organizing Into Suites at Test Runtime ==
	The stride test runner allows you to specify a "path" under which a ~~test unit's~~ results will be published. The path is a forward-slash delimited list of suite names. The path is optionally specified in the argument to the <tt>-r</tt> command line option.		The stride test runner allows you to specify a "path" under which a tests results will be published. The path is a forward-slash delimited list of suite names. The path is optionally specified in the argument to the <tt>-r</tt> command line option.

	Following are some examples. Many more examples can be seen [[Stride_Runner#Input\|here]]:.		Following are some examples. The examples shown apply to both host script test modules and target test units. Many more examples can be seen [[Stride_Runner#Input\|here]]:.
	~~;<tt><-r omitted></tt>~~		;<tt>-r /{Test1; Test2}</tt>
	~~: If you omit the -r argument, stride will run all of the test units specified in the database and publish them into a single top-level root test suite~~		: Here we specify two tests to be run and published to the root suite
	;<tt>-r /{~~TestUnit1~~; ~~TestUnit2~~}</tt>		;<tt>-r /suite1{Test1} -r /suite2{Test2}</tt>
	: Here we specify two ~~test units~~ to be run and published to the root suite		: Here the two tests are each published to their own suite. (Note that <tt>-r</tt> can be specified any number of times on a stride command line.)
	;<tt>-r /suite1{~~TestUnit1~~} -r /suite2{~~TestUnit2~~}</tt>		;<tt>-r /suite1/suite1.1/suite1.1.1{Test1; Test2}</tt>
	: Here the two ~~test units~~ are each published to their own suite. (Note that <tt>-r</tt> can be specified any number of times on a stride command line.)		: Here, the two tests are published to a suite that is several levels deep in a hierarchy. Note that stride will create suites as needed to fulfill your publishing requirements.
	;<tt>-r /suite1/suite1.1/suite1.1.1{~~TestUnit1~~; ~~TestUnit2~~}</tt>
	: Here, the two ~~test units~~ are published to a suite that is several levels deep in a hierarchy. Note that stride will create suites as needed to fulfill your publishing requirements.

	Another important benefit to be gained by organizing your ~~test units~~ into suites is that a subtotal roll-up of several important results are provided by each suite. Pass/fail counts, error counts, and time under tests are all summed at the suite level for all tests that are under the current suite and any of its subsuites.		Another important benefit to be gained by organizing your tests into suites is that a subtotal roll-up of several important results are provided by each suite. Pass/fail counts, error counts, and time under tests are all summed at the suite level for all tests that are under the current suite and any of its subsuites.

	== Techniques for Test Organization ==		== Techniques for Test Organization ==
Line 28:		Line 26:
	Another variation on this technique is to have each workgroup maintain their own options file, then submit all of the options files to stride when the aggregated tests are run. (You can specify more than one options file on the stride command line: e.g. <tt>-O group1.txt -O group2.txt ...</tt>.)		Another variation on this technique is to have each workgroup maintain their own options file, then submit all of the options files to stride when the aggregated tests are run. (You can specify more than one options file on the stride command line: e.g. <tt>-O group1.txt -O group2.txt ...</tt>.)

	===Automating Your Test Organization===		===Organize Test Modules in Directories ===
	~~The key to reliable and repeatable organization is through automation.~~ If you adopt a standard project-wide naming convention for your test units, a little bit of scripting glue can provide the organization by allowing test unit authors to determine where the test unit will be published within the suite hierarchy.		We recommend that you leverage host directory structures to organize your script based test modules. Entire directories of test modules can be executed (recursively searched) by passing the directory names to the <tt>--run</tt> option

			===Automating Your Test Unit Organization===
			If you adopt a standard project-wide naming convention for your native code test units, a little bit of scripting glue can provide the organization by allowing test unit authors to determine where the test unit will be published within the suite hierarchy.

	This technique works like this: at the time the tests are run, the available test unit names are listed, and from these names the path of parent suites is determined. We create an options file dynamically from this information and submit it to stride when tests are run.		This technique works like this: at the time the tests are run, the available test unit names are listed, and from these names the path of parent suites is determined. We create an options file dynamically from this information and submit it to stride when tests are run.
Line 47:		Line 48:
	Note that the full test unit name (in curly braces) must be specified since this is the name that is known to stride.		Note that the full test unit name (in curly braces) must be specified since this is the name that is known to stride.

	=== OrganizeIntoSuites.pl ===		==== OrganizeIntoSuites.pl ====
	The following perl script reads a list of test unit names from <tt>stdin</tt> and creates a "suite-ized" run command line parameter for each, writing to <tt>stdout</tt>.		The following perl script reads a list of test unit names from <tt>stdin</tt> and creates a "suite-ized" run command line parameter for each, writing to <tt>stdout</tt>.

Line 116:		Line 117:
	</pre>		</pre>

	=== Putting It All Together ===		==== Putting It All Together ====
	By combining the pre-processing of the test unit list with the running of the tests in a shell script or batch file, you can create a fully automated solution.		By combining the pre-processing of the test unit list with the running of the tests in a shell script or batch file, you can create a fully automated solution.