Expectations

Overview

STRIDE supports Behavior Testing, which is different than Unit Testing or API Testing in that it does not focus on calling functions and validating return values. Behavior Testing validates the expected sequencing and state of the software executing under normal operating conditions. In order to begin this type of testing, you must (1) instrument the source under test with Test Points and (2) define the expectations of the Test Points for each test scenario. To learn more about the uniqueness of Behavior Testing read here.

The Validation Model is based on validating that the Expected List of Test Points have been hit as well as optionally validating any state data associated with a Test Point member. Through the use of an Unexpected List, it's possible to simulataneously validate that specified Test Points are NOT hit during the execution of the scenario.

The Expected List and Unexpected List declared for a specific testing scenario define the active set of Test Points for the system. All other Test Points within the software are not active and have nominal impact. To learn more about testable software builds read here.

Expected List

The first step in defining Expectations is describing the set of Test Points expected to be hit during a test scenario. This is the list of Test Points and the expected number of hits (count) for each of the Test Points. Any expected state data associated with a Test Point that requires validation should be included.

The validation of the Expected List is done when any of the following conditions are met:

• The Expected List sequencing has been met completely
• The time allocated for validation has expired
• State data associated with a Test Point is declared invalid
• An out-of-sequence Test Point has been encountered (a concrete failure is identified)
• An Unexpected Test Point has been encountered

Expected TEST POINTS list:

Label	Count	Expected State Data
Name 1	1	<describe data payload validation requirements if applicable>
Name 2	1 +	<describe data payload validation requirements if applicable>
name ...	n	<...>

Sequencing Properties

The Expected List requires defining sequencing properties used for the validation. This involves establishing how to process the ordering of the Test Points being hit, how strict to handle duplications of Test Point members, and if to continue capturing Test Points for the entire time allocated for the test scenario even though the Expected List sequencing requirements have been met.

Expected TEST POINTS sequencing properties:

Properties	Description
Ordered	Test Points are expected to be hit in the exact order defined in the list
Unordered	Test Points can be hit in any order defined in the list
Strict	Test Points specified in the list must match the exact count (i.e. no duplication)
Non-Strict	Test Points specified in the list can be duplicated anywhere in the sequence
Continue	Test Points validation will continue even though the Expected List has been met so far

State Data Validation

State data validation for a Test Point member is optional. The Expected List can associate a predicate as a necessary condition for validation of a Test Point member when being processed. The predicate provides an extra level of validation (beyond sequencing) focusing on the state data associated with the Test Point.

The predicate can declare the Test Point:

• Valid - successful, thus the expected count is decremented
• Invalid - the expectations is NOT met, indicating failure
• Ignored - no affect on the validation, continue to wait on the current Test Point

Unexpected List

An Unexpected List of Test Points can optionally be defined. This list works in conjunction with the Expected List. The list defines a set of Test Points that are to be treated as failures if any one of them is encountered (hit) during a testing scenario.

This list can be made up of the complement of the Expected List, thus including all Test Points not included in that defined set. This type of setting requires enabling all Test Points defined in the software during the execution of the test.

If the Unexpected List members intersect with the Expected List members, that is considered an input error.

Special Processing

Members

There are two special members that can be used for the Expected List called ANY IN SET and ANY AT ALL. These special member values are used to customize the success criteria (i.e. trigger condition) for the entry defined within the Expected List. Both types require a predicate to determine if the Test Point is valid, invalid, or ignored. The ANY IN SET refers to the set of points defined by the Expected List while the ANY AT ALL refers to all Test Points in the system. By definition the ANY AT ALL member requires enabling all Test Points defined in the software during the execution of the test.

Concerning an Unordered Expected List, only one special member can be used. It will be processed first independent of its location within the list. The Count attribute is also optionally available to be used with the special members, indicating the number of valid returns from the predicate until the next Test Point in the list is validated.

There is one special member that can be used for the Unexpected List called EVERYTHING ELSE. This special member can only be used as a single member only within an Unexpected List. It takes the complement of the set of Test Points defined within the Expected List. This requires enabling all Test Points defined in the software during the execution of the test. Also, by definition any non-expected Test Point hit during the test would be deemed a failure.

Counts

There is one special count value that can be associated with a Test Point defined in the Expected List called ANY COUNT. This special count is used to focus on capturing a variable number of Test Point hits -- 0 or more. In essences the Test Point has no consequences of the Expected List sequencing, except if a predicate associated with the Test Point returns invalid.

Use Cases

The following use cases provide examples of defining Expectations based on different testing scenarios.

Sequencing Properties

Use case examples for validating the sequencing of Test Points.

Expecting an exact ordered sequence of Test Points to be hit

 Expect = [A,B,C,D]; ORDERED, STRICT
 Hits   = [A,B,C,D]   - PASS
 Hits   = [A,B,D,C]   - FAIL (D hit before C)

Expecting an exact ordered of Test Points but with duplicates

 Expect = [A,B,C,D]; ORDERED, STRICT
 Hits   = [A,B,B,C,D] - FAIL (B duplicated with strict defined)

 Expect = [A,B,C,D]; ORDERED, NON-STRICT
 Hits   = [A,B,B,C,D] - PASS
 Hits   = [A,C,B,C,D] - PASS
 Hits   = [A,B,C,B,D] - PASS

Expecting a set of Test Points to be hit, but the order does not matter

 Expect = [A,B,C,D]; UNORDERED, STRICT
 Hits   = [B,A,C,D]   - PASS
 Hits   = [B,A,C,B,D] - FAIL (B hit twice)

 Expect = [A,B,C,D]; UNORDERED, NON-STRICT
 Hits   = [B,A,C,B,D] - PASS 

Expecting a set of Test Points to be hit, but order and duplications does not matter

 Expect = [A,B,C,D]; UNORDERED, NON-STRICT
 Hits   = [B,A,C,B,D] - PASS
 Hits   = [B,A,A,A,D] - FAIL (C never hit)

Expecting an exact ordered sequence of Test Points to be hit, but checking for any extra hits for total duration of test scenario

 Expect = [A,B,C,D]; ORDERED, STRICT, CONTINUE
 Hits   = [A,B,C,D]   - PASS
 Hits   = [A,B,C,D,A] - FAIL (A hit after valid sequence met based on test timeout)

Count Attribute

Use case examples that leverage the Count attribute associated with a list member.

Expecting a sequence of Test Points, including multiple hits for 2 members

 Expect = [A,B:3,C,D:2]; ORDERED, STRICT
 Hits   = [A,B,B,B,C,D,D]   - PASS
 Hits   = [A,B,B,C,B,D,D]   - FAIL (B not hit three times before C)

 Expect = [A,B:3,C,D:2]; ORDERED, NON-STRICT
 Hits   = [A,B,B,B,C,B,D,D]  - PASS

Expecting a set of Test Points being hit, including multiple hits (with duplications), but order does not matter

 Expect = [A,B:3,C,D]; UNORDERED, STRICT
 Hits   = [B,A,B,C,B,D]     - PASS
 Hits   = [B,A,B,B,C,B,D]   - FAIL (B duplicated on 4th hit)

 Expect = [A,B:3,C,D]; UNORDERED, NON-STRICT
 Hits   = [B,A,B,B,C,B,D]   - PASS

State Data Validation

Use case examples leveraging predicates to validate state data.

Simple example showing associating a predicate with Test Point named A.

 Expect = [A:p1(tp),B,C,D]; ORDERED, STRICT
 p1(tp) { return VALID }
 Hits   = [A,B,C,D]   - PASS

Expecting Test Point B to be hit 3 times

 Expect = [A,B:3:p1(tp),C,D]; UNORDERED, STRICT
 p1(tp) { return VALID unless 3rd time called than return INVALID}
 Hits   = [B,A,B,C,B,D]     - FAIL (predicate returns invalid on 3rd B hit)

Special Members

Use case examples using the two special members within an Expected List.

Waiting on a single trigger (D) to start the processing of an Expected List

 Expect = [ANY_IN_SET:p1(tp),A,B,C:2,D]; ORDERED, STRICT
 p1(tp) {return VALID when D hit otherwise IGNORE}
 Hits   = [A,B,A,D*,A,B,C,C,D] - PASS

Waiting on Test Point outside of the Expected List to trigger processing

 Expect = [ANY_AT_ALL:p1(tp),A,B,C:2,D]; ORDERED, STRICT
 p1(tp) {return VALID when x hit}
 Hits   = [A,B,A,x*,A,B,C,C,D] - PASS

Waiting on a multiple triggers - 1st trigger used to start processing and 2nd used midstream

 Expect = [ANY_IN_SET:p1(tp),A,B,ANY_IN_SET:p2(tp),C:2,D,E]; ORDERED, STRICT
 p1(tp) {return VALID when D hit}
 p2(tp) {return VALID when E hit}
 Hits   = [A,B,A,D*,A,B,A,A,B,E*,C,C,D,E] - PASS

 Expect = [ANY_IN_SET:p1(tp),A,B,C:2,D]; UNORDERED, NON-STRICT
 p1(tp) {return VALID when D hit}
 Hits   = [A,B,A,D*,A,A,A,D,A,C,B,C,D] - PASS

This test uses the ANY AT ALL along with a fix count of 3

 Expect = [ANY_AT_ALL:p1(tp):3,A,B,C]; ORDERED, STRICT
 p1(tp) {return VALID}
 Hits   = [A,z,x,A,B,C]   - PASS

Special Counts

Use case examples leveraging the special count attributes within an Expected List.

Expecting between 0 and more Test Point As to be hit before processing the remaining members.

 Expect = [A:ANY_COUNT,B,C,D]; ORDERED, STRICT
 Hits   = [B,C,D]     - PASS
 Hits   = [A,A,B,C,D]   - PASS

This test passes immediately after the A is hit (example of a weird expectation)

 Expect = [A,B:ANY_COUNT]; ORDERED, STRICT
 Hits   = [A,..]      - PASS

This test also passes, but will process all Test Point B hits until the test time expires

 Expect = [A,B:ANY_COUNT]; ORDERED, STRICT, CONTINUE
 Hits   = [A,B,B,B]    - PASS (the wait timeout will complete the validation)
 Hits   = [A]          - PASS

This test validates a set of Test Points focusing exclusively on the associated state data

 Expect = [A:p1(tp):ANY_COUNT, B:p2(tp):ANY_COUNT, C:p2(tp):ANY_COUNT]; UNORDERED, CONTINUE (STRICT NA)
 Hits   = [A,A,B,A,C,C,C,A,..] - PASS (unless any of the predicates fails)
 Hits   = [A,B,C,D]            - PASS

Miscellaneous

The following are miscellaneous use cases.

Expecting an exact ordered sequence of Test Points to be hit, but verifying certain Test Points are NOT hit

 Expect   = [A,B,C,D]; ORDERED, STRICT
 Unexpect = [x,y]
 Hits     = [A,B,C,D]    - PASS
 Hits     = [A,B,B,C,D]  - FAIL
 Hits     = [A,B,C,y,..] - FAIL

This test will always pass and log every Test Point hit during test time duration. Note all Test Points are active.

 Expect = [ANY_AT_ALL:p1(tp):ANY_COUNT]; CONTINUE {other sequencing properties NA}
 p1(tp) {return VALID}
 Hits   = [A,A,B,x,y,D, ..]   - PASS (the wait timeout will complete the validation)

This test will log every Test Point hit during test, looking for X and Y as failures. Note all Test Points are active.

 Expect = [ANY_AT_ALL:p1(tp):ANY_COUNT]; CONTINUE {other sequencing properties NA}
 p1(tp) {return VALID}
 Unexpect = [X,Y]
 Hits   = [A,A,B,x,y,D,Y]   - FAIL (last Test Point hit causes a failure)

This test will fail on any Test Point that is hit. Note all Test Points are active. Test will last based on time duration or the 1st Test Point hit.

 Expect   = NULL; CONTINUE {other sequencing properties NA}
 Unexpect = [EVERYTHING_ELSE]
 Hits   = [A,..]   - FAIL 

This test will ignore any Test Point hit during test, looking for X and Y as failures.

 Expect   = NULL; CONTINUE {other sequencing properties NA}
 Unexpect = [X,Y]
 Hits   = [..,X,..]   - FAIL

This test will fail on any Test Points hit except for A & B. Note all Test Points are active. Test will last until failure or timeout.

 Expect   = [A,B]; UNORDERED, NON-STRICT, CONTINUE
 Unexpect = [EVERYTHING_ELSE]
 Hits   = [..,A,..,X]   - FAIL