Expectations - Revision history

Marku at 02:25, 7 July 2015

2015-07-07T02:25:51Z

← Older revision		Revision as of 02:25, 7 July 2015
Line 1:		Line 1:
	__NOTOC__		__NOTOC__
	~~STRIDE~~ supports '''Behavior Testing''', which is different than Unit Testing or API Testing in that it does not focus on calling functions and validating return values. Behavior Testing validates the ''expected sequencing and state of the software'' executing under normal operating conditions. In order to begin this type of testing, you must '''(1)''' instrument the source under test with [[Test Point \| Test Points]] and '''(2)''' define the ''expectations'' of the Test Points for each test scenario.		Stride supports '''Behavior Testing''', which is different than Unit Testing or API Testing in that it does not focus on calling functions and validating return values. Behavior Testing validates the ''expected sequencing and state of the software'' executing under normal operating conditions. In order to begin this type of testing, you must '''(1)''' instrument the source under test with [[Test Point \| Test Points]] and '''(2)''' define the ''expectations'' of the Test Points for each test scenario.

	The '''Validation Model''' is based on validating that the ''Expected List'' of Test Points have been hit as well as optionally validating any ''state data'' associated with a Test Point member. Through the use of an ''Unexpected List'', it's possible to simulataneously validate that specified Test Points are NOT hit during the execution of the scenario.		The '''Validation Model''' is based on validating that the ''Expected List'' of Test Points have been hit as well as optionally validating any ''state data'' associated with a Test Point member. Through the use of an ''Unexpected List'', it's possible to simulataneously validate that specified Test Points are NOT hit during the execution of the scenario.

Marku at 20:27, 6 July 2015

2015-07-06T20:27:31Z

← Older revision		Revision as of 20:27, 6 July 2015
Line 1:		Line 1:
	~~== Overview ==~~		__NOTOC__
			STRIDE supports '''Behavior Testing''', which is different than Unit Testing or API Testing in that it does not focus on calling functions and validating return values. Behavior Testing validates the ''expected sequencing and state of the software'' executing under normal operating conditions. In order to begin this type of testing, you must '''(1)''' instrument the source under test with [[Test Point \| Test Points]] and '''(2)''' define the ''expectations'' of the Test Points for each test scenario.
	STRIDE supports '''Behavior Testing''', which is different than Unit Testing or API Testing in that it does not focus on calling functions and validating return values. Behavior Testing validates the ''expected sequencing and state of the software'' executing under normal operating conditions. In order to begin this type of testing, you must '''(1)''' instrument the source under test with [[Test Point \| Test Points]] and '''(2)''' define the ''expectations'' of the Test Points for each test scenario~~. To learn more about the uniqueness of Behavior Testing [[What_is_Unique_About_STRIDE#STRIDE_includes_behavior-based_testing_techniques \| read here]]~~.

	The '''Validation Model''' is based on validating that the ''Expected List'' of Test Points have been hit as well as optionally validating any ''state data'' associated with a Test Point member. Through the use of an ''Unexpected List'', it's possible to simulataneously validate that specified Test Points are NOT hit during the execution of the scenario.		The '''Validation Model''' is based on validating that the ''Expected List'' of Test Points have been hit as well as optionally validating any ''state data'' associated with a Test Point member. Through the use of an ''Unexpected List'', it's possible to simulataneously validate that specified Test Points are NOT hit during the execution of the scenario.

	The ''Expected List'' and ''Unexpected List'' declared for a specific testing scenario define the ''active'' set of Test Points for the system. All other Test Points within the software are not active and have nominal impact~~. To learn more about testable software builds [[What_is_Unique_About_STRIDE#STRIDE_software_builds_are_both_functional_and_testable \| read here]]~~.		The ''Expected List'' and ''Unexpected List'' declared for a specific testing scenario define the ''active'' set of Test Points for the system. All other Test Points within the software are not active and have nominal impact.

	== Expected List ==		= Expected List =

	The first step in defining '''Expectations''' is describing the set of Test Points expected to be hit during a test scenario. This is the '''list''' of Test Points and the expected number of hits ('''count''') for each of the Test Points. Any expected '''state data''' associated with a Test Point that requires validation should be included.		The first step in defining '''Expectations''' is describing the set of Test Points expected to be hit during a test scenario. This is the '''list''' of Test Points and the expected number of hits ('''count''') for each of the Test Points. Any expected '''state data''' associated with a Test Point that requires validation should be included.
Line 77:		Line 76:
	* Ignored - no affect on the validation, continue to wait on the current Test Point		* Ignored - no affect on the validation, continue to wait on the current Test Point

	== Unexpected List ==		= Unexpected List =

	An ''Unexpected List'' of Test Points can optionally be defined. This list works in conjunction with the [[Expectations#Expected_List \| ''Expected List'']]. The list defines a set of Test Points that are to be treated as failures if any one of them is encountered (hit) during a testing scenario.		An ''Unexpected List'' of Test Points can optionally be defined. This list works in conjunction with the [[Expectations#Expected_List \| ''Expected List'']]. The list defines a set of Test Points that are to be treated as failures if any one of them is encountered (hit) during a testing scenario.
Line 85:		Line 84:
	If the ''Unexpected List'' members intersect with the ''Expected List'' members, that is considered an input error.		If the ''Unexpected List'' members intersect with the ''Expected List'' members, that is considered an input error.

	== Special Processing ==		= Special Processing =

	=== Members ===		=== Members ===
Line 97:		Line 96:
	There is one ''special count'' value that can be associated with a Test Point defined in the ''Expected List'' called '''ANY COUNT'''. This special count is used to focus on capturing a variable number of Test Point hits -- 0 or more. In essences the Test Point has no consequences of the Expected List sequencing, except if a predicate<ref name="predicate"/> associated with the Test Point returns ''invalid''.		There is one ''special count'' value that can be associated with a Test Point defined in the ''Expected List'' called '''ANY COUNT'''. This special count is used to focus on capturing a variable number of Test Point hits -- 0 or more. In essences the Test Point has no consequences of the Expected List sequencing, except if a predicate<ref name="predicate"/> associated with the Test Point returns ''invalid''.

	== Use Cases ==		<hr>


			= Use Cases =

	The following use cases provide examples of defining '''Expectations''' based on different testing scenarios.		The following use cases provide examples of defining '''Expectations''' based on different testing scenarios.
Line 265:		Line 267:
	== Notes ==		== Notes ==
	<references/>		<references/>


	~~[[Category: Source Instrumentation]]~~

Marku: /* Special Counts */

2012-02-16T01:04:17Z

Special Counts

← Older revision		Revision as of 01:04, 16 February 2012
Line 216:		Line 216:
	Hits = [A,B,B,B] - PASS (the wait timeout will complete the validation)		Hits = [A,B,B,B] - PASS (the wait timeout will complete the validation)
	Hits = [A] - PASS		Hits = [A] - PASS

			This test fails because all Test Points are checked once the test time expires
			''Expect'' = [A]; ''ORDERED, STRICT, CONTINUE''
			Hits = [A,A] - FAILS (the wait timeout than will complete the validation)

	This test validates a set of Test Points focusing exclusively on the associated ''state data''		This test validates a set of Test Points focusing exclusively on the associated ''state data''

Marku at 19:05, 1 November 2010

2010-11-01T19:05:46Z

Show changes

Timd: /* Sequencing Properties */

2010-08-25T00:55:54Z

Sequencing Properties

← Older revision		Revision as of 00:55, 25 August 2010
Line 106:		Line 106:
	Hits = [A,B,D,C] - FAIL (D hit before C)		Hits = [A,B,D,C] - FAIL (D hit before C)

	Expecting an exact ordered of Test Points but with duplicates		Expecting an exact ordered sequence of Test Points but with duplicates
	''Expect'' = [A,B,C,D]; ''ORDERED, STRICT''		''Expect'' = [A,B,C,D]; ''ORDERED, STRICT''
	Hits = [A,B,B,C,D] - FAIL (B duplicated with strict defined)		Hits = [A,B,B,C,D] - FAIL (B duplicated with strict defined)

	''Expect'' = [A,B,C,D]; ''ORDERED, NON-STRICT''		''Expect'' = [A,B,C,D]; ''ORDERED, NON-STRICT''
	Hits = [A,B,B,C,D] - PASS		Hits = [A,B,B,C,D] - PASS (A,B,C,D seen in order, but with extra B)
	Hits = [A,C,B,C,D] - PASS		Hits = [A,C,B,C,D] - PASS (A,B,C,D seen in order, but with extra C)
	Hits = [A,B,C,B,D] - PASS		Hits = [A,B,C,B,D] - PASS (A,B,C,D seen in order, but with extra B)

	Expecting a set of Test Points to be hit, but the order does not matter		Expecting a set of Test Points to be hit, but the order does not matter
Line 121:		Line 121:

	''Expect'' = [A,B,C,D]; ''UNORDERED, NON-STRICT''		''Expect'' = [A,B,C,D]; ''UNORDERED, NON-STRICT''
	Hits = [B,A,C,B,D] - PASS		Hits = [B,A,C,B,D] - PASS (All expected points hit at least once)

	Expecting a set of Test Points to be hit, but order and duplications does not matter		Expecting a set of Test Points to be hit, but order and duplications does not matter
	''Expect'' = [A,B,C,D]; ''UNORDERED, NON-STRICT''		''Expect'' = [A,B,C,D]; ''UNORDERED, NON-STRICT''
	Hits = [B,A,C,B,D] - PASS		Hits = [B,A,C,B,D] - PASS (All expected points hit at least once)
	Hits = [B,A,A,A,D] - FAIL (C never hit)		Hits = [B,A,A,A,D] - FAIL (C never hit; this will fail by timeout wait expiration)

	=== Count Attribute ===		=== Count Attribute ===

Ivailop: /* Members */

2010-07-14T20:38:09Z

Members

← Older revision		Revision as of 20:38, 14 July 2010
Line 88:		Line 88:
	Concerning an '''Unordered''' ''Expected List'', only one special member can be used. It will be processed first independent of its location within the list. The '''Count''' attribute is also optionally available to be used with the special members, indicating the number of valid returns from the predicate until the next Test Point in the list is validated.		Concerning an '''Unordered''' ''Expected List'', only one special member can be used. It will be processed first independent of its location within the list. The '''Count''' attribute is also optionally available to be used with the special members, indicating the number of valid returns from the predicate until the next Test Point in the list is validated.

	There is one ''special member'' that can be used for the ''Unexpected List'' called '''EVERYTHING ELSE'''. This special member can only be used as a '''single member only''' within an ''Unexpected List''. It takes the ''complement'' of the set of Test Points defined within the ''Expected List''. This requires enabling all Test Points defined in the software during the execution of the test. Also, by definition any Test Point hit during the test would be deemed a failure.		There is one ''special member'' that can be used for the ''Unexpected List'' called '''EVERYTHING ELSE'''. This special member can only be used as a '''single member only''' within an ''Unexpected List''. It takes the ''complement'' of the set of Test Points defined within the ''Expected List''. This requires enabling all Test Points defined in the software during the execution of the test. Also, by definition any non-expected Test Point hit during the test would be deemed a failure.

	=== Counts ===		=== Counts ===

Ivailop: /* Special Counts */

2010-07-14T20:06:05Z

Special Counts

← Older revision		Revision as of 20:06, 14 July 2010
Line 193:		Line 193:
	''Expect'' = [A,B:ANY_COUNT]; ''ORDERED, STRICT''		''Expect'' = [A,B:ANY_COUNT]; ''ORDERED, STRICT''
	Hits = [A,...] - PASS		Hits = [A,...] - PASS
	~~Hits = [A,B,B,...] - PASS~~

	=== Miscellaneous ===		=== Miscellaneous ===

Ivailop: /* Special Members */

2010-07-14T19:05:19Z

Special Members

← Older revision		Revision as of 19:05, 14 July 2010
Line 177:		Line 177:
	Hits = [A,B,A,D,A,B,A,A,B,E,C,C,D,E] - PASS		Hits = [A,B,A,D,A,B,A,A,B,E,C,C,D,E] - PASS

	''Expect'' = [~~ANY_IN_SET~~:p1(tp),A,B,C:2,D]; ''UNORDERED, NON-STRICT''		''Expect'' = [ANYTHING:p1(tp),A,B,C:2,D]; ''UNORDERED, NON-STRICT''
	p1(tp) {return VALID when D hit}		p1(tp) {return VALID when D hit}
	Hits = [A,B,A,D*,A,A,A,D,A,C,B,C,D] - PASS		Hits = [A,B,A,D*,A,A,A,D,A,C,B,C,D] - PASS


	=== Special Counts ===		=== Special Counts ===

Ivailop: /* Special Counts */

2010-07-14T19:00:35Z

Special Counts

← Older revision		Revision as of 19:00, 14 July 2010
Line 188:		Line 188:
	Expecting between 0 and more Test Point As to be hit before processing the remaining members.		Expecting between 0 and more Test Point As to be hit before processing the remaining members.
	''Expect'' = [A:ANY_COUNT,B,C,D]; ''ORDERED, STRICT''		''Expect'' = [A:ANY_COUNT,B,C,D]; ''ORDERED, STRICT''
	Hits = [B,C,D] - PASS		Hits = [B,C,D] - PASS
	Hits = [A,B,C,D] - PASS		Hits = [A,A,B,C,D] - PASS

	This test passes ''immediately'' after the A is hit (example of a weird expectation)		This test passes ''immediately'' after the A is hit (example of a weird expectation)
	''Expect'' = [A,B:ANY_COUNT]; ''ORDERED, STRICT''		''Expect'' = [A,B:ANY_COUNT]; ''ORDERED, STRICT''
	Hits = [A,..] - PASS		Hits = [A,...] - PASS
			Hits = [A,B,B,...] - PASS

	=== Miscellaneous ===		=== Miscellaneous ===

Mikee: /* Members */

2010-07-13T22:25:24Z

Members

← Older revision		Revision as of 22:25, 13 July 2010
Line 84:		Line 84:

	=== Members ===		=== Members ===
	There is one ''special member'' that can be used for the ''Expected List'' called '''~~ANYTHNING~~'''. This special member value is used to customize the success criteria (i.e. trigger condition) for the entry defined within the ''Expected List''. It requires a predicate<ref name="predicate"/> to determine if the Test Point is ''valid, invalid, or ignored''. The '''ANYTHING''' refers to the set of points defined by the ''Expected List''.		There is one ''special member'' that can be used for the ''Expected List'' called '''ANYTHING'''. This special member value is used to customize the success criteria (i.e. trigger condition) for the entry defined within the ''Expected List''. It requires a predicate<ref name="predicate"/> to determine if the Test Point is ''valid, invalid, or ignored''. The '''ANYTHING''' refers to the set of points defined by the ''Expected List''.

	Concerning an '''Unordered''' ''Expected List'', only one special member can be used. It will be processed first independent of its location within the list. The '''Count''' attribute is also optionally available to be used with the special members, indicating the number of valid returns from the predicate until the next Test Point in the list is validated.		Concerning an '''Unordered''' ''Expected List'', only one special member can be used. It will be processed first independent of its location within the list. The '''Count''' attribute is also optionally available to be used with the special members, indicating the number of valid returns from the predicate until the next Test Point in the list is validated.