Source Instrumentation Overview - Revision history

Marku: /* Instrumentation */

2010-12-09T15:50:11Z

Instrumentation

← Older revision		Revision as of 15:50, 9 December 2010
Line 15:		Line 15:

	== Instrumentation ==		== Instrumentation ==
	To make the software ''testable'', the first step in the process is for the experts to selectively insert instrumentation macros -- called [[Test_Point \| Test Points]] -- into the source code. Test Points themselves have nominal impact on the performance of the application – they are only active during test data collection<ref name="n1"> Test data collection is typically implemented in a low priority background thread. The data is captured in the calling routine's thread context (no context switch) but processed in the background or on the host. Instrumentation macros return immediately to the caller (i.e. no-op) when testing is not active</ref>. Test Points contain names and optional payload data. When Test Points are activated, they are collected in the background, along with timing and any associated data. The set of Test Points hit, their order, timing, and data content can all be used to validate that the software is behaving as expected. [[Test_Log \| Test Logs]] can also be added to the source code to provide additional information in the context of an executing test.		To make the software ''testable'', the first step in the process is for the experts to selectively insert instrumentation macros -- called [[Test_Point \| Test Points]] -- into the source code. Test Points themselves have nominal impact on the performance of the application – they are only active during test data collection<ref name="n1"> Test data collection is typically implemented in a low priority background thread. The data is captured in the calling routine's thread context (no context switch) but processed in the background or on the host. Instrumentation macros return immediately to the caller (i.e. no-op) when testing is not active.</ref>. Test Points contain names and optional payload data. When Test Points are activated, they are collected in the background, along with timing and any associated data. The set of Test Points hit, their order, timing, and data content can all be used to validate that the software is behaving as expected. [[Test_Log \| Test Logs]] can also be added to the source code to provide additional information in the context of an executing test.

	Here are the requirements for source instrumentation using the STRIDE Framework:		Here are the requirements for source instrumentation using the STRIDE Framework:

Marku: /* Instrumentation */

2010-12-09T15:49:28Z

Instrumentation

← Older revision		Revision as of 15:49, 9 December 2010
Line 15:		Line 15:

	== Instrumentation ==		== Instrumentation ==
	To make the software ''testable'', the first step in the process is for the experts to selectively insert instrumentation macros -- called [[Test_Point \| Test Points]] -- into the source code. Test Points themselves have nominal impact on the performance of the application – they are only active during test data collection<ref name="n1"> Test data collection is typically implemented in a low priority background thread. The data is captured in the calling routine's thread context (no context switch) but processed in the background or on the host. Instrumentation macros return immediately to the caller (i.e. ~~NOP~~)~~</ref>~~ when testing is not active. Test Points contain names and optional payload data. When Test Points are activated, they are collected in the background, along with timing and any associated data. The set of Test Points hit, their order, timing, and data content can all be used to validate that the software is behaving as expected. [[Test_Log \| Test Logs]] can also be added to the source code to provide additional information in the context of an executing test.		To make the software ''testable'', the first step in the process is for the experts to selectively insert instrumentation macros -- called [[Test_Point \| Test Points]] -- into the source code. Test Points themselves have nominal impact on the performance of the application – they are only active during test data collection<ref name="n1"> Test data collection is typically implemented in a low priority background thread. The data is captured in the calling routine's thread context (no context switch) but processed in the background or on the host. Instrumentation macros return immediately to the caller (i.e. no-op) when testing is not active</ref>. Test Points contain names and optional payload data. When Test Points are activated, they are collected in the background, along with timing and any associated data. The set of Test Points hit, their order, timing, and data content can all be used to validate that the software is behaving as expected. [[Test_Log \| Test Logs]] can also be added to the source code to provide additional information in the context of an executing test.

	Here are the requirements for source instrumentation using the STRIDE Framework:		Here are the requirements for source instrumentation using the STRIDE Framework:

Mikee at 23:51, 11 June 2010

2010-06-11T23:51:07Z

← Older revision		Revision as of 23:51, 11 June 2010
Line 15:		Line 15:

	== Instrumentation ==		== Instrumentation ==
	To make the software ''testable'' the first step is the process is for the experts to selectively insert instrumentation macros called [[Test_Point \| Test ~~Point~~]] into the source code. Test Points themselves have nominal impact on the performance of the application – they are only active during test data collection<ref name="n1"> Test data collection is typically implemented in a low priority background thread. The data is captured in the calling routine's thread context (no context switch) but processed in the background. ~~When testing is not active instrumentation~~ macros return immediately to the caller (i.e. NOP)</ref>. Test Points ~~contains~~ names and ~~optionally~~ payload data. When Test Points are activated, they are collected in the background, along with timing and any associated data. The set of Test Points hit, their order, timing, and data content can all be used to validate the software is behaving as expected. [[Test_Log \| Test Logs]] can also be added to the source code to provide additional information in the context of an executing test.		To make the software ''testable'', the first step in the process is for the experts to selectively insert instrumentation macros -- called [[Test_Point \| Test Points]] -- into the source code. Test Points themselves have nominal impact on the performance of the application – they are only active during test data collection<ref name="n1"> Test data collection is typically implemented in a low priority background thread. The data is captured in the calling routine's thread context (no context switch) but processed in the background or on the host. Instrumentation macros return immediately to the caller (i.e. NOP)</ref> when testing is not active. Test Points contain names and optional payload data. When Test Points are activated, they are collected in the background, along with timing and any associated data. The set of Test Points hit, their order, timing, and data content can all be used to validate that the software is behaving as expected. [[Test_Log \| Test Logs]] can also be added to the source code to provide additional information in the context of an executing test.

	Here are the requirements for source instrumentation using the STRIDE Framework:		Here are the requirements for source instrumentation using the STRIDE Framework:
Line 24:		Line 24:

	== Expectations ==		== Expectations ==
	In addition to instrumenting the source code with Test Points, defining the [[Expectations]] of the Test Points is required for testing. The involves defining the list of Test Points expected to be hit during a given test scenario. Also included is any expected '''data''' associated with a Test Point that requires validation.

	~~== Testings ==~~		In addition to instrumenting the source code with Test Points, you must also define the [[Expectations]] of the Test Points. This involves defining the list of Test Points expected to be hit during a given test scenario. An expectation can also include any expected '''data''' associated with a Test Point that requires validation.
	~~Once~~ the source ~~under test has been instrumented and~~ the [[Expectations ~~\| expectations~~]] ~~defined, STRIDE offers a number~~ of ~~techniques~~ to be ~~used for implementing~~ '''~~expectation tests~~'''. For non-developers the [[Test_Modules_Overview \| STRIDE Scripting Solution]] is recommended. Scripting allows testers to leverage the power of dynamic languages that execute on the host. The framework provides script libraries that ~~automate the behavior validation as well as hooks for customization. Also there are minimal software build dependencies using scripting for~~ validation. Scripting can also leverage [[Function_Capturing \| function remoting]] to fully automate test execution using script modules [[Perl_Script_Snippets#Invoking_a_function_on_the_target \| invoking functions on the target.]]

	For developers writing expectation tests the [[Test_Units_Overview \| STRIDE Test Units]] ~~and implementation~~ in [[Expectation_Tests_in_C/C%2B%2B \| C or C++]] is recommended ~~to begin with~~.		== Testing ==

			Once the source under test has been instrumented and the [[Expectations \| expectations]] defined, STRIDE offers a number of techniques that can be used for implementing '''expectation tests'''. For non-developers the [[Test_Modules_Overview \| STRIDE Scripting Solution]] is recommended. Scripting allows testers to leverage the power of dynamic languages that execute on the host. The framework provides script libraries that automate the behavior validation as well as hooks for customization. The script implementation also reduces software build dependencies since the test code is not part of the device image. Scripting can also leverage [[Function_Capturing \| function remoting]] to fully automate test execution by [[Perl_Script_Snippets#Invoking_a_function_on_the_target \| invoking functions on the target.]]

			For developers writing expectation tests, the [[Test_Units_Overview \| STRIDE Test Units]] with test logic implemented in [[Expectation_Tests_in_C/C%2B%2B \| C or C++]] is recommended as a starting point.

	== Notes ==		== Notes ==

Mikee: /* Introduction */

2010-06-11T23:43:08Z

Introduction

← Older revision		Revision as of 23:43, 11 June 2010
Line 1:		Line 1:
	== Introduction ==		== Introduction ==
	Source instrumentation is the process by which developers and domain experts selectively instrument the source under test for the purpose of writing test scenarios against the executing application. Implementing tests that ~~leverages~~ source instrumentation is called '''Expectation Testing'''. This validation technique is very useful for verifying proper code sequencing based on the software's internal design.		Source instrumentation is the process by which developers and domain experts selectively instrument the source under test for the purpose of writing test scenarios against the executing application. Implementing tests that leverage source instrumentation is called '''Expectation Testing'''. This validation technique is very useful for verifying proper code sequencing based on the software's internal design.

	Unlike traditional unit testing that drives testing based on input parameters and isolating functionality, expectation testing is executed within a fully functional software build running on a real target platform. Expectation tests are not dependent on ~~passed in~~ input parameters, but often ~~leverages~~ the same types of input / output controls used by functional and black-box testing.		Unlike traditional unit testing that drives testing based on input parameters and isolating functionality, expectation testing is executed within a fully functional software build running on a real target platform. Expectation tests are not dependent on input parameters, but often leverage the same types of input/output controls used by functional and black-box testing.

	Another unique feature of expectation testing is that ~~being a~~ domain ~~expert~~ is not required to implement a test. Developers and domain experts use instrumentation to export design knowledge of the software to the entire team. ~~Also~~ there is no stubbing required, no special logic to ~~generated~~ input parameters, and ~~even~~ knowledge of how the application software is coded ~~or built is not necessarily required~~.		Another unique feature of expectation testing is that domain expertise is not required to implement a test. Developers and domain experts use instrumentation to export design knowledge of the software to the entire team. Furthermore, there is no stubbing required, no special logic to generate input parameters, and no advanced knowledge required of how the application software is coded.

	To enable effective test coverage developers and domain experts are required to insert instrumentation at key locations to gain insight and testability. Here are some general suggested source code areas to consider instrumenting:		To enable effective test coverage, developers and domain experts are required to insert instrumentation at key locations to gain insight and testability. Here are some general suggested source code areas to consider instrumenting:
	* critical function entry/exit points		* critical function entry/exit points
	* state transitions		* state transitions

Marku: /* Implementing Tests */

2010-05-12T16:22:24Z

Implementing Tests

← Older revision		Revision as of 16:22, 12 May 2010
Line 26:		Line 26:
	In addition to instrumenting the source code with Test Points, defining the [[Expectations]] of the Test Points is required for testing. The involves defining the list of Test Points expected to be hit during a given test scenario. Also included is any expected '''data''' associated with a Test Point that requires validation.		In addition to instrumenting the source code with Test Points, defining the [[Expectations]] of the Test Points is required for testing. The involves defining the list of Test Points expected to be hit during a given test scenario. Also included is any expected '''data''' associated with a Test Point that requires validation.

	== ~~Implementing Tests~~ ==		== Testings ==
	Once the source under test has been instrumented and the ~~expectation requirements~~ defined, STRIDE offers a number of techniques to be used for implementing '''expectation tests'''. For non-developers the [[Test_Modules_Overview \| STRIDE Scripting Solution]] is recommended. Scripting allows testers to leverage the power of dynamic languages that execute on the host. The framework provides script libraries that automate the behavior validation as well as hooks for customization. Also there are minimal software build dependencies using scripting for validation. Scripting can also leverage [[Function_Capturing \| function remoting]] to fully automate test execution using script modules [[Perl_Script_Snippets#Invoking_a_function_on_the_target \| invoking functions on the target.]]		Once the source under test has been instrumented and the [[Expectations \| expectations]] defined, STRIDE offers a number of techniques to be used for implementing '''expectation tests'''. For non-developers the [[Test_Modules_Overview \| STRIDE Scripting Solution]] is recommended. Scripting allows testers to leverage the power of dynamic languages that execute on the host. The framework provides script libraries that automate the behavior validation as well as hooks for customization. Also there are minimal software build dependencies using scripting for validation. Scripting can also leverage [[Function_Capturing \| function remoting]] to fully automate test execution using script modules [[Perl_Script_Snippets#Invoking_a_function_on_the_target \| invoking functions on the target.]]

	For developers writing expectation tests the [[Test_Units_Overview \| STRIDE Test Units]] and implementation in [[Expectation_Tests_in_C/C%2B%2B \| C or C++]] is recommended to begin with.		For developers writing expectation tests the [[Test_Units_Overview \| STRIDE Test Units]] and implementation in [[Expectation_Tests_in_C/C%2B%2B \| C or C++]] is recommended to begin with.

Marku: /* Defining Expectations */

2010-05-12T16:17:29Z

Defining Expectations

@@ Line 23: / Line 23: @@
 * selectively instrument strategic locations in your source code with [[Test_Point | Test Points]] and optionally [[Test_Log | Test Logs]]
-== Defining Expectations ==
+== Expectations ==
-In addition to instrumenting the source code with Test Points, defining the expectation requirements based on testing scenarios is required. The first step is defining the set of Test Points expected to be hit during a test scenario. This is the '''list''' of Test Points and the expected number of hits for each of the Test Points. Also included is any expected '''data''' associated with a Test Point that requires validation.
+In addition to instrumenting the source code with Test Points, defining the [[Expectations]] of the Test Points is required for testing. The involves defining the list of Test Points expected to be hit during a given test scenario. Also included is any expected '''data''' associated with a Test Point that requires validation.
 == Implementing Tests ==

Marku at 21:36, 11 May 2010

2010-05-11T21:36:10Z

@@ Line 1: / Line 1: @@
-Source instrumentation is the process by which developers and domain experts selectively instrument the source under test for the purpose of writing test scenarios against the executing application. Here are the requirements for source instrumentation using the STRIDE Framework:
+== Introduction ==
-* define the '''STRIDE_ENABLED''' preprocessor macro in your build system.
+Unlike traditional unit testing that drives testing based on input parameters and isolating functionality, expectation testing is executed within a fully functional software build running on a real target platform. Expectation tests are not dependent on passed in input parameters, but often leverages the same types of input / output controls used by functional and black-box testing.
-The process of instrumenting the source under test requires domain expertise to maximize the benefit of instrumentation. Here are some general suggested source code areas to consider instrumenting to gain insight and testability:
+Another unique feature of expectation testing is that being a domain expert is not required to implement a test. Developers and domain experts use instrumentation to export design knowledge of the software to the entire team. Also there is no stubbing required, no special logic to generated input parameters, and even knowledge of how the application software is coded or built is not necessarily required.
 * critical function entry/exit points
 * state transitions
 * critical or interesting data transitions (using optional payload to convey data values)
 [[Category: Source Instrumentation]]

Ivailop: moved Test Point Overview to Source Instrumentation Overview

2010-02-16T22:55:31Z

moved Test Point Overview to Source Instrumentation Overview

← Older revision	Revision as of 22:55, 16 February 2010
(No difference)

Marku: Created page with 'Source instrumentation is the process by which developers and domain experts selectively instrument the source under test for the purpose of writing test scenarios against the ex…'

2010-01-22T00:22:23Z

Created page with 'Source instrumentation is the process by which developers and domain experts selectively instrument the source under test for the purpose of writing test scenarios against the ex…'

New page

Source instrumentation is the process by which developers and domain experts selectively instrument the source under test for the purpose of writing test scenarios against the executing application. Here are the requirements for source instrumentation using the STRIDE Framework:

* define the '''STRIDE_ENABLED''' preprocessor macro in your build system.
* include the '''srtest.h''' header file. This file included in the STRIDE Runtime source distribution.
* selectively instrument strategic locations in your source code with [[Test Point]] or [[Test Log]] macros.

The process of instrumenting the source under test requires domain expertise to maximize the benefit of instrumentation. Here are some general suggested source code areas to consider instrumenting to gain insight and testability:
* critical function entry/exit points
* high value and exceptional callbacks
* state transitions
* critical or interesting data transitions (using optional payload to convey data values)

[[Category: Source Instrumentation]]