Test Point - Revision history

Jeffs: Update implementation step titles

2015-07-21T17:21:26Z

Update implementation step titles

← Older revision		Revision as of 17:21, 21 July 2015
Line 36:		Line 36:

	The steps required to implement an Expectation test are the following:		The steps required to implement an Expectation test are the following:
	#[[#~~Instrumentation~~ \| ~~Instrumentation~~]]		#[[#Instrument Code \| Instrument Code]]
	#[[#Define ~~your~~ Expectations \| Define ~~your~~ Expectations]]		#[[#Define Expectations \| Define Expectations]]
	#[[#Write ~~the~~ Test ~~Unit~~ \| Write ~~the~~ Test ~~Unit~~]]		#[[#Write Test Units \| Write Test Units]]

	== ~~Instrumentation~~ ==		== Instrument Code ==
	To make the software ''testable'', the first step in the process is for the experts to selectively insert instrumentation macros into the source code. Test Points have a nominal impact on the performance of the application as they are only active during test data collection<ref name="n1"> Test data collection is typically implemented in a low priority background thread. The data is captured in the calling routine's thread context (no context switch) but processed in the background or on the host. Instrumentation macros return immediately to the caller (i.e. no-op) when testing is not active.</ref>. Test Points contain names and optional payload data. When Test Points are activated, they are collected in the background, along with timing and any associated data. The set of Test Points hit, their order, timing, and data content can all be used to validate that the software is behaving as expected. [[Test_Log \| Test Logs]] can also be added to the source code to provide additional information in the context of an executing test.		To make the software ''testable'', the first step in the process is for the experts to selectively insert instrumentation macros into the source code. Test Points have a nominal impact on the performance of the application as they are only active during test data collection<ref name="n1"> Test data collection is typically implemented in a low priority background thread. The data is captured in the calling routine's thread context (no context switch) but processed in the background or on the host. Instrumentation macros return immediately to the caller (i.e. no-op) when testing is not active.</ref>. Test Points contain names and optional payload data. When Test Points are activated, they are collected in the background, along with timing and any associated data. The set of Test Points hit, their order, timing, and data content can all be used to validate that the software is behaving as expected. [[Test_Log \| Test Logs]] can also be added to the source code to provide additional information in the context of an executing test.

Line 65:		Line 65:
	\|}		\|}

	== Define ~~your~~ Expectations ==		== Define Expectations ==

	In addition to instrumenting the source code with Test Points, you must also define the [[Expectations]] of the Test Points. This involves defining the list of Test Points expected to be hit during a given test scenario. Expectations can also include any '''data''' associated with a Test Point that requires validation.		In addition to instrumenting the source code with Test Points, you must also define the [[Expectations]] of the Test Points. This involves defining the list of Test Points expected to be hit during a given test scenario. Expectations can also include any '''data''' associated with a Test Point that requires validation.

	== Write ~~the~~ Test ~~Unit~~ ==		== Write Test Units ==
	Once the source under test has been instrumented and the [[Expectations]] defined, Stride offers a number of techniques that can be used for implementing '''expectation tests'''.		Once the source under test has been instrumented and the [[Expectations]] defined, Stride offers a number of techniques that can be used for implementing '''expectation tests'''.

Jeffs at 17:16, 21 July 2015

2015-07-21T17:16:24Z

← Older revision		Revision as of 17:16, 21 July 2015
Line 23:		Line 23:


	Unlike traditional unit testing that drives testing based on input parameters and isolating functionality, ''Expectation'' testing is executed within a fully functional software build running on a real target platform. Test Points are not dependent on input parameters, but often leverage the same types of input/output controls used by functional and black-box testing.		Unlike traditional unit testing that drives testing based on input parameters and isolating functionality, Expectation testing is executed within a fully functional software build running on a real target platform. Test Points are not dependent on input parameters, but often leverage the same types of input/output controls used by functional and black-box testing.

	Another unique feature of this type of testing is that domain expertise is not required to implement a test. Developers and domain experts use instrumentation to export design knowledge of the software to the entire team. Furthermore, ~~there is~~ no stubbing required, no special logic to generate input parameters, and no advanced knowledge required of how the application software is coded.		Another unique feature of this type of testing is that domain expertise is not required to implement a test. Developers and domain experts use instrumentation to export design knowledge of the software to the entire team. Furthermore, no stubbing is required, no special logic to generate input parameters, and no advanced knowledge required of how the application software is coded.

	To enable effective test coverage, developers and domain experts ~~are required to~~ insert instrumentation at key locations to gain insight and testability. Here are some general suggested source code areas to consider instrumenting:		To enable effective test coverage, developers and domain experts insert instrumentation at key locations to gain insight and testability. Here are some general suggested source code areas to consider instrumenting:
	* entry/exit points of critical functions		* entry/exit points of critical functions
	* state transitions		* state transitions
Line 41:		Line 41:

	== Instrumentation ==		== Instrumentation ==
	To make the software ''testable'', the first step in the process is for the experts to selectively insert instrumentation macros into the source code. Test Points ~~themselves~~ have nominal impact on the performance of the application – they are only active during test data collection<ref name="n1"> Test data collection is typically implemented in a low priority background thread. The data is captured in the calling routine's thread context (no context switch) but processed in the background or on the host. Instrumentation macros return immediately to the caller (i.e. no-op) when testing is not active.</ref>. Test Points contain names and optional payload data. When Test Points are activated, they are collected in the background, along with timing and any associated data. The set of Test Points hit, their order, timing, and data content can all be used to validate that the software is behaving as expected. [[Test_Log \| Test Logs]] can also be added to the source code to provide additional information in the context of an executing test.		To make the software ''testable'', the first step in the process is for the experts to selectively insert instrumentation macros into the source code. Test Points have a nominal impact on the performance of the application as they are only active during test data collection<ref name="n1"> Test data collection is typically implemented in a low priority background thread. The data is captured in the calling routine's thread context (no context switch) but processed in the background or on the host. Instrumentation macros return immediately to the caller (i.e. no-op) when testing is not active.</ref>. Test Points contain names and optional payload data. When Test Points are activated, they are collected in the background, along with timing and any associated data. The set of Test Points hit, their order, timing, and data content can all be used to validate that the software is behaving as expected. [[Test_Log \| Test Logs]] can also be added to the source code to provide additional information in the context of an executing test.

	To specify a ~~test point~~ you should include the '''srtest.h''' header file from the Stride Runtime in your compilation unit. The Test Point macros are active only when <tt>STRIDE_ENABLED</tt> is <tt>#define</tt>d, therefore it is practical to place these macros in-line in production source. When <tt>STRIDE_ENABLED</tt> is not <tt>#define</tt>d, these macros evaluate to nothing.		To specify a Test Point you should include the '''srtest.h''' header file from the Stride Runtime in your compilation unit. The Test Point macros are active only when <tt>STRIDE_ENABLED</tt> is <tt>#define</tt>d, therefore it is practical to place these macros in-line in production source. When <tt>STRIDE_ENABLED</tt> is not <tt>#define</tt>d, these macros evaluate to nothing.

	{\| class="prettytable"		{\| class="prettytable"
Line 67:		Line 67:
	== Define your Expectations ==		== Define your Expectations ==

	In addition to instrumenting the source code with Test Points, you must also define the [[Expectations]] of the Test Points. This involves defining the list of Test Points expected to be hit during a given test scenario. ~~Expectation~~ can also include any '''data''' associated with a Test Point that requires validation.		In addition to instrumenting the source code with Test Points, you must also define the [[Expectations]] of the Test Points. This involves defining the list of Test Points expected to be hit during a given test scenario. Expectations can also include any '''data''' associated with a Test Point that requires validation.

	== Write the Test Unit ==		== Write the Test Unit ==

Jeffs at 21:05, 8 July 2015

2015-07-08T21:05:24Z

← Older revision		Revision as of 21:05, 8 July 2015
Line 28:		Line 28:

	To enable effective test coverage, developers and domain experts are required to insert instrumentation at key locations to gain insight and testability. Here are some general suggested source code areas to consider instrumenting:		To enable effective test coverage, developers and domain experts are required to insert instrumentation at key locations to gain insight and testability. Here are some general suggested source code areas to consider instrumenting:
	* ~~critical function~~ entry/exit points		* entry/exit points of critical functions
	* state transitions		* state transitions
	* critical or interesting data transitions (using optional payload to convey data values)		* critical or interesting data transitions (using optional payload to convey data values)

Jeffs: /* Instrumentation */

2015-07-08T20:59:06Z

Instrumentation

← Older revision		Revision as of 20:59, 8 July 2015
Line 43:		Line 43:
	To make the software ''testable'', the first step in the process is for the experts to selectively insert instrumentation macros into the source code. Test Points themselves have nominal impact on the performance of the application – they are only active during test data collection<ref name="n1"> Test data collection is typically implemented in a low priority background thread. The data is captured in the calling routine's thread context (no context switch) but processed in the background or on the host. Instrumentation macros return immediately to the caller (i.e. no-op) when testing is not active.</ref>. Test Points contain names and optional payload data. When Test Points are activated, they are collected in the background, along with timing and any associated data. The set of Test Points hit, their order, timing, and data content can all be used to validate that the software is behaving as expected. [[Test_Log \| Test Logs]] can also be added to the source code to provide additional information in the context of an executing test.		To make the software ''testable'', the first step in the process is for the experts to selectively insert instrumentation macros into the source code. Test Points themselves have nominal impact on the performance of the application – they are only active during test data collection<ref name="n1"> Test data collection is typically implemented in a low priority background thread. The data is captured in the calling routine's thread context (no context switch) but processed in the background or on the host. Instrumentation macros return immediately to the caller (i.e. no-op) when testing is not active.</ref>. Test Points contain names and optional payload data. When Test Points are activated, they are collected in the background, along with timing and any associated data. The set of Test Points hit, their order, timing, and data content can all be used to validate that the software is behaving as expected. [[Test_Log \| Test Logs]] can also be added to the source code to provide additional information in the context of an executing test.

	To specify a test point you should include the '''srtest.h''' header file from the ~~STRIDE~~ Runtime in your compilation unit. The Test Point macros are active only when <tt>STRIDE_ENABLED</tt> is <tt>#define</tt>d, therefore it is practical to place these macros in-line in production source. When <tt>STRIDE_ENABLED</tt> is not <tt>#define</tt>d, these macros evaluate to nothing.		To specify a test point you should include the '''srtest.h''' header file from the Stride Runtime in your compilation unit. The Test Point macros are active only when <tt>STRIDE_ENABLED</tt> is <tt>#define</tt>d, therefore it is practical to place these macros in-line in production source. When <tt>STRIDE_ENABLED</tt> is not <tt>#define</tt>d, these macros evaluate to nothing.

	{\| class="prettytable"		{\| class="prettytable"

Marku: /* Write the Test Unit */

2015-07-07T16:04:30Z

Write the Test Unit

← Older revision		Revision as of 16:04, 7 July 2015
Line 72:		Line 72:
	Once the source under test has been instrumented and the [[Expectations]] defined, Stride offers a number of techniques that can be used for implementing '''expectation tests'''.		Once the source under test has been instrumented and the [[Expectations]] defined, Stride offers a number of techniques that can be used for implementing '''expectation tests'''.

	* Tests can be written in [[~~Expectation_Tests_in_C~~/C%2B%2B \| C or C++]] and executed on the device under test using the Stride framework.		* Tests can be written in [[Test_Point_Testing_in_C/C%2B%2B \| C or C++]] and executed on the device under test using the Stride framework.
	* Tests can be written in [[Perl_Script_APIs \| Perl]].		* Tests can be written in [[Perl_Script_APIs \| Perl]].

	== Notes ==		== Notes ==
	<references/>		<references/>

Marku: /* Write the Test Unit */

2015-07-06T20:23:07Z

Write the Test Unit

← Older revision		Revision as of 20:23, 6 July 2015
Line 70:		Line 70:

	== Write the Test Unit ==		== Write the Test Unit ==
	Once the source under test has been instrumented and the [[Expectations ~~\| expectations~~]] defined, Stride offers a number of techniques that can be used for implementing '''expectation tests'''.		Once the source under test has been instrumented and the [[Expectations]] defined, Stride offers a number of techniques that can be used for implementing '''expectation tests'''.

	* Tests can be written in [[Expectation_Tests_in_C/C%2B%2B \| C or C++]] and executed on the device under test using the Stride framework.		* Tests can be written in [[Expectation_Tests_in_C/C%2B%2B \| C or C++]] and executed on the device under test using the Stride framework.
	* Tests can be written in [[~~Test_Modules_Overview~~ \| Perl ~~Test Scripts~~]]		* Tests can be written in [[Perl_Script_APIs \| Perl]].

	== Notes ==		== Notes ==
	<references/>		<references/>

Marku: Marku moved page Test Point2 to Test Point

2015-07-06T19:40:33Z

Marku moved page Test Point2 to Test Point

← Older revision	Revision as of 19:40, 6 July 2015
(No difference)

Marku: /* Write the Test Unit */

2015-07-02T17:49:48Z

Write the Test Unit

← Older revision		Revision as of 17:49, 2 July 2015
Line 70:		Line 70:

	== Write the Test Unit ==		== Write the Test Unit ==
			Once the source under test has been instrumented and the [[Expectations \| expectations]] defined, Stride offers a number of techniques that can be used for implementing '''expectation tests'''.

	~~Once the source under test has been instrumented and the [[Expectations \| expectations]] defined, STRIDE offers a number of techniques that~~ can be used for implementing '''expectation tests'''. For non-developers the [[Test_Modules_Overview \| STRIDE Scripting Solution]] is recommended. Scripting allows testers to leverage the power of dynamic languages that execute on the host. The framework provides script libraries that automate the behavior validation as well as hooks for customization. The script implementation also reduces software build dependencies since the test code is not part of the device image. Scripting can also leverage [[Function_Capturing \| function remoting]] to fully automate test execution by [[Perl_Script_Snippets#Invoking_a_function_on_the_target \| invoking functions on the target.]]		* Tests can be written in [[Expectation_Tests_in_C/C%2B%2B \| C or C++]] and executed on the device under test using the Stride framework.
			* Tests can be written in [[Test_Modules_Overview \| Perl Test Scripts]]
	~~For developers writing expectation tests, the [[Test_Units_Overview \| STRIDE Test Units]] with test logic implemented~~ in [[Expectation_Tests_in_C/C%2B%2B \| C or C++]] ~~is recommended as a starting point~~.


	== Notes ==		== Notes ==
	<references/>		<references/>

Marku: /* Define your Expectations */

2015-07-02T17:42:20Z

Define your Expectations

← Older revision		Revision as of 17:42, 2 July 2015
Line 67:		Line 67:
	== Define your Expectations ==		== Define your Expectations ==

	In addition to instrumenting the source code with Test Points, you must also define the [[Expectations]] of the Test Points. This involves defining the list of Test Points expected to be hit during a given test scenario. ~~An expectation~~ can also include any ~~expected~~ '''data''' associated with a Test Point that requires validation.		In addition to instrumenting the source code with Test Points, you must also define the [[Expectations]] of the Test Points. This involves defining the list of Test Points expected to be hit during a given test scenario. Expectation can also include any '''data''' associated with a Test Point that requires validation.

	== Write the Test Unit ==		== Write the Test Unit ==

Marku: Created page with "NOTOC Source instrumentation is the process by which developers and domain experts selectively instrument the source under test for the purpose of writing test scenarios a..."

2015-07-02T17:23:07Z

Created page with "__NOTOC__ Source instrumentation is the process by which developers and domain experts selectively instrument the source under test for the purpose of writing test scenarios a..."

New page

__NOTOC__
Source instrumentation is the process by which developers and domain experts selectively instrument the source under test for the purpose of writing test scenarios against the executing application. Implementing tests that leverage source instrumentation is called '''Expectation Testing'''. This validation technique is very useful for verifying proper code sequencing based on the software's internal design.

<source lang='c'>
#include <srtest.h>
...
/* a test point with no payload */
srTEST_POINT("first test point");

/* a test point with binary payload */
srTEST_POINT_DATA("second test point", myData, sizeofMyData);

/* a test point with simple string payload */
srTEST_POINT_STR("third test point", "payload with simple string");

/* a test point with formatted string payload */
srTEST_POINT_STR("third test point", "payload with format string %d", myVar);

#ifdef __cplusplus
srTEST_POINT_STR("c++ test point", "") << "stream input supported under c++";
#endif
</source>

Unlike traditional unit testing that drives testing based on input parameters and isolating functionality, ''Expectation'' testing is executed within a fully functional software build running on a real target platform. Test Points are not dependent on input parameters, but often leverage the same types of input/output controls used by functional and black-box testing.

Another unique feature of this type of testing is that domain expertise is not required to implement a test. Developers and domain experts use instrumentation to export design knowledge of the software to the entire team. Furthermore, there is no stubbing required, no special logic to generate input parameters, and no advanced knowledge required of how the application software is coded.

To enable effective test coverage, developers and domain experts are required to insert instrumentation at key locations to gain insight and testability. Here are some general suggested source code areas to consider instrumenting:
* critical function entry/exit points
* state transitions
* critical or interesting data transitions (using optional payload to convey data values)
* callback routines
* data persistence
* error conditions

The steps required to implement an Expectation test are the following:
#[[#Instrumentation | Instrumentation]]
#[[#Define your Expectations | Define your Expectations]]
#[[#Write the Test Unit | Write the Test Unit]]

== Instrumentation ==
To make the software ''testable'', the first step in the process is for the experts to selectively insert instrumentation macros into the source code. Test Points themselves have nominal impact on the performance of the application – they are only active during test data collection<ref name="n1"> Test data collection is typically implemented in a low priority background thread. The data is captured in the calling routine's thread context (no context switch) but processed in the background or on the host. Instrumentation macros return immediately to the caller (i.e. no-op) when testing is not active.</ref>. Test Points contain names and optional payload data. When Test Points are activated, they are collected in the background, along with timing and any associated data. The set of Test Points hit, their order, timing, and data content can all be used to validate that the software is behaving as expected. [[Test_Log | Test Logs]] can also be added to the source code to provide additional information in the context of an executing test.

To specify a test point you should include the '''srtest.h''' header file from the STRIDE Runtime in your compilation unit. The Test Point macros are active only when <tt>STRIDE_ENABLED</tt> is <tt>#define</tt>d, therefore it is practical to place these macros in-line in production source. When <tt>STRIDE_ENABLED</tt> is not <tt>#define</tt>d, these macros evaluate to nothing.

{| class="prettytable"
| colspan="2" | '''Test Point Macros'''
|-valign="top"
| '''srTEST_POINT'''(''label'')
| ''label'' is a pointer to a null-terminated string

|-valign="top"
| '''srTEST_POINT_DATA'''(''label'', ''data'', ''size'')
| ''label'' is a pointer to a null-terminated string 
''data'' is a pointer to a byte sequence 
''size'' is the size of the ''data'' in bytes

|-valign="top"
| '''srTEST_POINT_STR'''(''label'', ''message'')
| ''label'' is a pointer to a null-terminated string 
''message'' is a pointer to a null-terminated format string 
''...'' variable list matching the format string 
When used in the context of a c++ compilation unit, this macro also supports the streaming operator to append to the message string (see example below)
|}

== Define your Expectations ==

In addition to instrumenting the source code with Test Points, you must also define the [[Expectations]] of the Test Points. This involves defining the list of Test Points expected to be hit during a given test scenario. An expectation can also include any expected '''data''' associated with a Test Point that requires validation.

== Write the Test Unit ==

Once the source under test has been instrumented and the [[Expectations | expectations]] defined, STRIDE offers a number of techniques that can be used for implementing '''expectation tests'''. For non-developers the [[Test_Modules_Overview | STRIDE Scripting Solution]] is recommended. Scripting allows testers to leverage the power of dynamic languages that execute on the host. The framework provides script libraries that automate the behavior validation as well as hooks for customization. The script implementation also reduces software build dependencies since the test code is not part of the device image. Scripting can also leverage [[Function_Capturing | function remoting]] to fully automate test execution by [[Perl_Script_Snippets#Invoking_a_function_on_the_target | invoking functions on the target.]]

For developers writing expectation tests, the [[Test_Units_Overview | STRIDE Test Units]] with test logic implemented in [[Expectation_Tests_in_C/C%2B%2B | C or C++]] is recommended as a starting point.

== Notes ==
<references/>

Test Point - Revision history

Jeffs: Update implementation step titles

Jeffs at 17:16, 21 July 2015

Jeffs at 21:05, 8 July 2015

Jeffs: /* Instrumentation */

Marku: /* Write the Test Unit */

Marku: /* Write the Test Unit */

Marku: Marku moved page Test Point2 to Test Point

Marku: /* Write the Test Unit */

Marku: /* Define your Expectations */

Marku: Created page with "__NOTOC__ Source instrumentation is the process by which developers and domain experts selectively instrument the source under test for the purpose of writing test scenarios a..."

Marku: Created page with "NOTOC Source instrumentation is the process by which developers and domain experts selectively instrument the source under test for the purpose of writing test scenarios a..."