Expectations
Overview
STRIDE supports Behavior Testing, which is different than Unit Testing or API Testing in that it does not focus on calling functions and validating return values. Behavior Testing validates the expected sequencing and state of the software executing under normal operating conditions. In order to begin this type of testing, you must (1) instrument the source under test with Test Points and (2) define the expectations of the Test Points for each test scenario. To learn more about the uniqueness of Behavior Testing read here.
The Validation Model is based on validating that the Expected List of Test Points have been hit as well as optionally validating any state data associated with a Test Point member. Through the use of an Unexpected List, it's possible to simulataneously validate that specified Test Points are NOT hit during the execution of the scenario.
The Expected List and Unexpected List declared for a specific testing scenario define the active set of Test Points for the system. All other Test Points within the software are not active and have nominal impact. To learn more about testable software builds read here.
Expected List
The first step in defining Expectations is describing the set of Test Points expected to be hit during a test scenario. This is the list of Test Points and the expected number of hits (count) for each of the Test Points. Any expected state data associated with a Test Point that requires validation should be included.
The validation of the Expected List is done when any of the following conditions are met:
- The Expected List sequencing has been met completely
- The time allocated for validation has expired
- State data associated with a Test Point is declared invalid
- An out-of-sequence Test Point has been encountered (a concrete failure is identified)
- An Unexpected Test Point has been encountered
Expected TEST POINTS list:
| Label | Count | Expected State Data |
| Name 1 | 1 | <describe data payload validation requirements if applicable> |
| Name 2 | 1 + | <describe data payload validation requirements if applicable> |
| name ... | n | <...> |
Sequencing Properties
The Expected List requires defining sequencing properties used for the validation. This involves establishing how to process the ordering of the Test Points being hit and how strict to handle duplications of Test Point members.
Expected TEST POINTS sequencing properties:
| Properties | Description |
| Ordered | Test Points are expected to be hit in the exact order defined in the list |
| Unordered | Test Points can be hit in any order defined in the list |
| Strict | Test Points specified in the list must match the exact count (i.e. no duplication) |
| Non-Strict | Test Points specified in the list can be duplicated anywhere in the sequence |
State Data Validation
State data validation for a Test Point member is optional. The Expected List can associate a predicate[1] as a necessary condition for validation of a Test Point member when being processed. The predicate provides an extra level of validation (beyond sequencing) focusing on the state data associated with the Test Point.
The predicate can declare the Test Point:
- Valid - successful, thus the expected count is decremented
- Invalid - the expectations is NOT met, indicating failure
- Ignored - no affect on the validation, continue to wait on the current Test Point
Unexpected List
An Unexpected List of Test Points can optionally be defined. This list works in conjunction with the Expected List. The list defines a set of Test Points that are to be treated as failures if any one of them is encountered (hit) during a testing scenario.
This list can be made up of the complement of the Expected List, thus including all Test Points not included in that defined set. This type of setting requires enabling all Test Points defined in the software during the execution of the test.
If the Unexpected List members intersect with the Expected List members, that is considered an input error.
Special Processing
Members
There is one special member that can be used for the Expected List called ANYTHING. This special member value is used to customize the success criteria (i.e. trigger condition) for the entry defined within the Expected List. It requires a predicate[1] to determine if the Test Point is valid, invalid, or ignored. The ANYTHING refers to the set of points defined by the Expected List.
Concerning an Unordered Expected List, only one special member can be used. It will be processed first independent of its location within the list. The Count attribute is also optionally available to be used with the special members, indicating the number of valid returns from the predicate until the next Test Point in the list is validated.
There is one special member that can be used for the Unexpected List called EVERYTHING ELSE. This special member can only be used as a single member only within an Unexpected List. It takes the complement of the set of Test Points defined within the Expected List. This requires enabling all Test Points defined in the software during the execution of the test. Also, by definition any non-expected Test Point hit during the test would be deemed a failure.
Counts
There is one special count value that can be associated with a Test Point defined in the Expected List called ANY COUNT. This special count is used to focus on capturing a variable number of Test Point hits -- 0 or more. In essences the Test Point has no consequences of the Expected List sequencing, except if a predicate[1] associated with the Test Point returns invalid.
Use Cases
The following use cases provide examples of defining Expectations based on different testing scenarios.
Sequencing Properties
Use case examples for validating the sequencing of Test Points.
Expecting an exact ordered sequence of Test Points to be hit
Expect = [A,B,C,D]; ORDERED, STRICT Hits = [A,B,C,D] - PASS Hits = [A,B,D,C] - FAIL (D hit before C)
Expecting an exact ordered sequence of Test Points but with duplicates
Expect = [A,B,C,D]; ORDERED, STRICT Hits = [A,B,B,C,D] - FAIL (B duplicated with strict defined)
Expect = [A,B,C,D]; ORDERED, NON-STRICT Hits = [A,B,B,C,D] - PASS (A,B,C,D seen in order, but with extra B) Hits = [A,C,B,C,D] - PASS (A,B,C,D seen in order, but with extra C) Hits = [A,B,C,B,D] - PASS (A,B,C,D seen in order, but with extra B)
Expecting a set of Test Points to be hit, but the order does not matter
Expect = [A,B,C,D]; UNORDERED, STRICT Hits = [B,A,C,D] - PASS Hits = [B,A,C,B,D] - FAIL (B hit twice)
Expect = [A,B,C,D]; UNORDERED, NON-STRICT Hits = [B,A,C,B,D] - PASS (All expected points hit at least once)
Expecting a set of Test Points to be hit, but order and duplications does not matter
Expect = [A,B,C,D]; UNORDERED, NON-STRICT Hits = [B,A,C,B,D] - PASS (All expected points hit at least once) Hits = [B,A,A,A,D] - FAIL (C never hit; this will fail by timeout wait expiration)
Count Attribute
Use case examples that leverage the Count attribute associated with a list member.
Expecting a sequence of Test Points, including multiple hits for 2 members
Expect = [A,B:3,C,D:2]; ORDERED, STRICT Hits = [A,B,B,B,C,D,D] - PASS Hits = [A,B,B,C,B,D,D] - FAIL (B not hit three times before C)
Expect = [A,B:3,C,D:2]; ORDERED, NON-STRICT Hits = [A,B,B,B,C,B,D,D] - PASS
Expecting a set of Test Points being hit, including multiple hits (with duplications), but order does not matter
Expect = [A,B:3,C,D]; UNORDERED, STRICT Hits = [B,A,B,C,B,D] - PASS Hits = [B,A,B,B,C,B,D] - FAIL (B duplicated on 4th hit)
Expect = [A,B:3,C,D]; UNORDERED, NON-STRICT Hits = [B,A,B,B,C,B,D] - PASS
State Data Validation
Use case examples leveraging predicates[1] to validate state data.
Simple example showing associating a predicate with Test Point named A.
Expect = [A:p1(tp),B,C,D]; ORDERED, STRICT
p1(tp) { return VALID }
Hits = [A,B,C,D] - PASS
Expecting Test Point B to be hit 3 times
Expect = [A,B:3:p1(tp),C,D]; UNORDERED, STRICT
p1(tp) { return VALID unless 3rd time called than return INVALID}
Hits = [B,A,B,C,B,D] - FAIL (predicate returns invalid on 3rd B hit)
Special Members
Use case examples using the two special members within an Expected List.
Waiting on a single trigger (D) to start the processing of an Expected List
Expect = [ANYTHING:p1(tp),A,B,C:2,D]; ORDERED, STRICT
p1(tp) {return VALID when D hit otherwise IGNORE}
Hits = [A,B,A,D*,A,B,C,C,D] - PASS
Waiting on a multiple triggers - 1st trigger used to start processing and 2nd used midstream
Expect = [ANYTHING:p1(tp),A,B,ANYTHING:p2(tp),C:2,D,E]; ORDERED, STRICT
p1(tp) {return VALID when D hit}
p2(tp) {return VALID when E hit}
Hits = [A,B,A,D*,A,B,A,A,B,E*,C,C,D,E] - PASS
Expect = [ANYTHING:p1(tp),A,B,C:2,D]; UNORDERED, NON-STRICT
p1(tp) {return VALID when D hit}
Hits = [A,B,A,D*,A,A,A,D,A,C,B,C,D] - PASS
Special Counts
Use case examples leveraging the special count attributes within an Expected List.
Expecting between 0 and more Test Point As to be hit before processing the remaining members.
Expect = [A:ANY_COUNT,B,C,D]; ORDERED, STRICT Hits = [B,C,D] - PASS Hits = [A,A,B,C,D] - PASS
This test passes immediately after the A is hit (example of a weird expectation)
Expect = [A,B:ANY_COUNT]; ORDERED, STRICT Hits = [A,...] - PASS
Miscellaneous
The following are miscellaneous use cases.
Expecting an exact ordered sequence of Test Points to be hit, but verifying certain Test Points are NOT hit
Expect = [A,B,C,D]; ORDERED, STRICT Unexpect = [x,y] Hits = [A,B,C,D] - PASS Hits = [A,B,B,C,D] - FAIL Hits = [A,B,C,y,..] - FAIL