Test Point Testing in C/C++: Difference between revisions

From STRIDE Wiki
Jump to navigation Jump to search
m (Replace STRIDE with Stride)
 
(9 intermediate revisions by 3 users not shown)
Line 1: Line 1:
== Introduction ==
__NOTOC__
Test Point [[Expectations]] tests can be written in native code (C/C++) and executed on the device under test using the STRIDE framework. In some cases, creating tests that run on the device itself might be preferable to host/script based tests, particularly if dealing with complex binary data payloads from test points.
 
== Instrumenting Source Code ==
Before any useful tests can be written, the source under test must be strategically instrumented with STRIDE Test Point macros. See [[Test_Point|this article]] for information on how to accomplish that.
 
== Creating Test Point Tests ==
Test code to validate the [[Expectations]] of the Test Points often follows this basic pattern:
Test code to validate the [[Expectations]] of the Test Points often follows this basic pattern:


# Specify an expectation set consisting of expected (i.e. the test points that are expected to be hit) and optionally unexpected (i.e. the test points that are not expected to be hit) test points
# Specify an expectation set consisting of expected (i.e. the test points that are expected to be hit) and optionally unexpected (i.e. the test points that are not expected to be hit) test points
# Register the expectation set with the STRIDE runtime
# Register the expectation set with the Stride runtime
# Invoke the software under test (causing instrumentation points to be hit). This may not be necessary if the instrumented software under test is constantly running).
# Invoke the software under test (causing instrumentation points to be hit). This may not be necessary if the instrumented software under test is constantly running).
# Wait for the expectation set to be satisfied or a timeout to occur
# Wait for the expectation set to be satisfied or a timeout to occur
Line 33: Line 27:
         {0}};
         {0}};


   /* register the expectation set with the STRIDE */
   /* register the expectation set*/
   srWORD handle;
   srWORD handle;
   srTestPointSetup(expected, unexpected, srTEST_POINT_EXPECT_UNORDERED, srTEST_CASE_DEFAULT, &handle);
   srTestPointSetup(expected, unexpected, srTEST_POINT_EXPECT_UNORDERED, srTEST_CASE_DEFAULT, &handle);
Line 48: Line 42:
#endif
#endif
</source>
</source>


== Reference ==
== Reference ==
Line 76: Line 71:
* A ''count'' value of either 0 or 1 is interpreted as 1  
* A ''count'' value of either 0 or 1 is interpreted as 1  
* The ''count'' could be set as "0 or more" by using the [[Expectations#Special_Processing|special]] srTEST_POINT_ANY_COUNT symbolic constant   
* The ''count'' could be set as "0 or more" by using the [[Expectations#Special_Processing|special]] srTEST_POINT_ANY_COUNT symbolic constant   
* A ''predicate'' value 0 indicates that any associated with a test point payload will be ignored.
* A ''predicate'' value 0 indicates that any associated data with a test point payload will be ignored.
* A ''user'' value 0 indicates that there is no user data associated with this test point
* A ''user'' value 0 indicates that there is no user data associated with this test point
* The ''label'' could be specified to ''any test point'' within the current universe by using the [[Expectations#Special_Processing|special]] srTEST_POINT_ANYTHING symbolic constant.  
* The ''label'' could be specified to ''any test point'' within the current expected set of test points by using the [[Expectations#Special_Processing|special]] srTEST_POINT_ANY_IN_SET or srTEST_POINT_ANY_AT_ALL symbolic constants.


==== Unexpected Array ====
==== Unexpected Array ====
Line 99: Line 94:


<source lang="c">
<source lang="c">
typedef srBYTE (*srTestPointPredicate_t)(const srTestPoint_t* ptTP, void* pvUser);
extern "C" typedef srBYTE (*srTestPointPredicate_t)(const srTestPoint_t* ptTP, void* pvUser);
</source>
</source>


Line 125: Line 120:


'''NOTE:'''
'''NOTE:'''
* As part of the standard STRIDE distribution there are three predefined function predicate helpers:
* As part of the standard Stride distribution there are three predefined function predicate helpers:
** srTestPointMemCmp - byte comparison
** srTestPointMemCmp - byte comparison
** srTestPointStrCmp - string case sensitive comparison
** srTestPointStrCmp - string case sensitive comparison
Line 159: Line 154:
srTEST_POINT_EXPECT_ORDERED - the test points are expected to be hit exactly in the defined order <br/>
srTEST_POINT_EXPECT_ORDERED - the test points are expected to be hit exactly in the defined order <br/>
srTEST_POINT_EXPECT_UNORDERED - the test points could to be hit in any order <br/>
srTEST_POINT_EXPECT_UNORDERED - the test points could to be hit in any order <br/>
srTEST_POINT_EXPECT_STRICT - the test points are expected to be hit exactly as specified <br/>
srTEST_POINT_EXPECT_STRICT - the test points are expected to be hit exactly as specified (no consecutive duplicate hits)<br/>
srTEST_POINT_EXPECT_NONSTRICT - other test points from the universe could to be hit in between
srTEST_POINT_EXPECT_NONSTRICT - other test points from the universe could to be hit in between <br/>
srTEST_POINT_EXPECT_CONTINUE - on successful expectation satisfaction continue processing until the wait timeout expires
|-
|-
| tTestCase   
| tTestCase   
Line 178: Line 174:
| srTRUE on success, srFALSE otherwise.
| srTRUE on success, srFALSE otherwise.
|}
|}
=== Expectation Validation ===
The behavior of the test point processing is as follows:
* If an unexpected test point is seen under test point wait, the expectation is failed and the wait is abandoned immediately.
* If an expected test point is seen under test point wait, the behavior varies depending on whether the expectation with the matching label includes a predicate function.
** Without Predicate, the expectation is unconditionally considered “hit,” and the expected count is decremented. If the expected count reaches zero, the expectation is considered satisfied. (In other words, the behavior is identical to an expectation with a predicate where the predicate returns srTRUE.)
** With Predicate, the predicate function determines the behavior depending on its return value. The predicate function is passed a copy of the data supplied with the original test point hit (along with other information) with which to determine its desired behavior. Depending on the return value:
*** srTRUE – the corresponding expectation is considered satisfied
*** srFALSE – the test point wait is abandoned and test case associated with the wait is set to fail status
*** srIGNORE – the test point hit is ignored
* If the expectation is not satisfied for the specified timeout period the expectation is considered failed. This rule has several exceptions:
** [[#Negative_Expectations|Negative Expectations]], don't expect some test points to be hit.
** [[#System_Observation|System Observation]], want to collect all test points.


==== srTestPointWait ====
==== srTestPointWait ====
Line 223: Line 206:


'''NOTES:'''
'''NOTES:'''
* The test thread blocks until either the expectation set is satisfied or the timeout elapses.
* The test thread blocks until either the expectation set is satisfied, unless srTEST_POINT_EXPECT_CONTINUE is specified on setup, or the timeout elapses.
* All test points hit during the wait (both expected and unexpected) are added to the test report as testcase comments
* All test points hit during the wait (both expected and unexpected) are added to the test report as testcase comments
* Once the wait is over (whether the expectation set has been satisfied or there has been a test failure), the current expectation set is automatically unregistered from the STRIDE runtime and the handle is released
* Once the wait is over (whether the expectation set has been satisfied or there has been a test failure), the current expectation set is automatically unregistered from the Stride runtime and the handle is released
* If you want to return immediately from a test case if expectation fails then make the check/wait call an argument to <tt>[[Test_Code_Macros#Assertions|srASSERT_TRUE()]]</tt>.
* If you want to return immediately from a test case if expectation fails then make the check/wait call an argument to the ''srASSERT_TRUE()'' macro.


==== srTestPointCheck ====
==== srTestPointCheck ====
Line 255: Line 238:
'''NOTES:'''
'''NOTES:'''
* All test points hit before the check (both expected and unexpected) are added to the test report as testcase comments
* All test points hit before the check (both expected and unexpected) are added to the test report as testcase comments
* Once the check is done (whether the expectation set has been satisfied or there has been a test failure), the current expectation set is automatically unregistered from the STRIDE runtime and the handle is released
* Once the check is done (whether the expectation set has been satisfied or there has been a test failure), the current expectation set is automatically unregistered from the Stride runtime and the handle is released
* If you want to return immediately from a test case if expectation fails then make the check/wait call an argument to <tt>[[Test_Code_Macros#Assertions|srASSERT_TRUE()]]</tt>.
* If you want to return immediately from a test case if expectation fails then make the check/wait call an argument to the ''srASSERT_TRUE()'' macro.  


=== C++ Facade Class ===
=== C++ Facade Class ===
The ''srtest.h'' file provides a simple [http://en.wikipedia.org/wiki/Facade_pattern facade] class that wraps the test point APIs described above in a simple C++ class called '''srTestPointsHandler'''. If you are writing your unit tests in C++ (using STRIDE test classes), then this class is available for your use. The class implements the following methods, which correspond exactly to the C API equivalents:
The ''srtest.h'' file provides a simple [http://en.wikipedia.org/wiki/Facade_pattern facade] class that wraps the test point APIs described above in a simple C++ class called '''srTestPointsHandler'''. If you are writing your unit tests in C++ (using Stride test classes), then this class is available for your use. The class implements the following methods, which correspond exactly to the C API equivalents:


; srTestPointsHandler : constructor. Takes four arguments which match exactly the first four parameters of the [[#srTestPointSetup|srTestPointSetup]] function.
; srTestPointsHandler : constructor. Takes four arguments which match exactly the first four parameters of the [[#srTestPointSetup|srTestPointSetup]] function.
; Wait : instance method that provides same functionality as [[#srTestPointWait|srTestPointWait]].  
; Wait : instance method that provides same functionality as [[#srTestPointWait|srTestPointWait]].  
; Check : instance method that provides same functionality as [[#srTestPointCheck|srTestPointCheck]].
; Check : instance method that provides same functionality as [[#srTestPointCheck|srTestPointCheck]].
== Use Cases ==
=== Known Expecations ===
In reality the amount of Test Points in an application under test is mutch larger then the number of test points of interest. Basically, the known expectation set is limited and consists of {A, B, C, D, X, Y, Z}, where some are expected and other unexpected, and everything else is unknown and should be ignored. In other words:
{| class="prettytable"
| '''Expected'''
| '''Unexpected'''
| '''Ignored'''
|-
| A, B, C, D
| X, Y, Z
| "everything else"
|}
To express that do the following:
<source lang="c">
#include <srtest.h>
void tf_testpoint_known(void)
{
  srTestPointExpect_t expected[]= {
        {"A"},
        {"B"},
        {"C"},
        {"D"},
        {0}};
  srTestPointUnexpect_t unexpected[]= {
        {"X"},
        {"Y"},
        {"Z"},
        {0}};
...
}
</source>
=== Full Expectations ===
If the whole application under test is known, the expectation set consists of all test points, where a limited set is expected and consists of {A, B, C, D} and the rest are unexpected, then nothing should be ignored. In other words:
{| class="prettytable"
| '''Expected'''
| '''Unexpected'''
| '''Ignored'''
|-
| A, B, C, D
| "everything else"
|
|}
To express that use the special <tt>srTEST_POINT_EVERYTHING_ELSE</tt> constant and do the following:
<source lang="c">
#include <srtest.h>
void tf_testpoint_all(void)
{
  srTestPointExpect_t expected[]= {
        {"A"},
        {"B"},
        {"C"},
        {"D"},
        {0}};
  srTestPointUnexpect_t unexpected[]= {
        {srTEST_POINT_EVERYTHING_ELSE},
        {0}};
...
}
</source>
=== 0 or More Expectations ===
In some cases the expected test point pattern is something like:
* START
* PROGRESS
...
* END
where any number (0 or more) of PROGRESS are expected but doesn't matter how many.
To specify that use the special <tt>srTEST_POINT_ANY_COUNT</tt> constant:
<source lang="c">
#include <srtest.h>
void tf_testpoint_any(void)
{
  srTestPointExpect_t expected[]= {
        {"START"},
        {"PROGRESS", srTEST_POINT_ANY_COUNT},
        {"END"},
        {0}};
...
}
</source>
=== Payload Expectations ===
In some cases the expected test point may carry a payload that needs to be validated against a user defined data:
* START
* PROGRESS_BIN({1,2,3})
* PROGRESS_STR("abc")
* END
where PROGRESS_BIN is expected to carry a binary payload and PROGRESS_STR - string payload.
To express that specify a predicate and user defined data:
<source lang="c">
#include <srtest.h>
void tf_testpoint_payload(void)
{
  srBYTE data[] = {1, 2, 3};
  srTestPointExpect_t expected[]= {
        {"START"},
        {"PROGRESS_BIN", 1, srTestPointMemCmp, data},
        {"PROGRESS_STR", 1, srTestPointStrCmp, "abc"},
        {"END"},
        {0}};
...
}
</source>
=== Negative Expectations ===
'''Case 1.''' Need to run a scenario and verify that NONE of the test points are hit.
Basically in that case everything is unexpected:
{| class="prettytable"
| '''Expected'''
| '''Unexpected'''
| '''Ignored'''
|-
|
| "everything"
|
|}
To express that use the special srTEST_POINT_EVERYTHING_ELSE constant and do the following:
<source lang="c">
#include <srtest.h>
void tf_testpoint_none(void)
{
  srTestPointExpect_t* expected = srNULL;
  srTestPointUnexpect_t unexpected[] = {
        {srTEST_POINT_EVERYTHING_ELSE},
        {0}};
...
}
</source>
'''Case 2.''' Need to verify that a subset of test points are not hit and everything else should be ignored.
Basically in that case a limited set of test points is unexpected:
{| class="prettytable"
| '''Expected'''
| '''Unexpected'''
| '''Ignored'''
|-
|
| X, Z, Y
| "everything else"
|}
<source lang="c">
#include <srtest.h>
void tf_testpoint_some(void)
{
  srTestPointExpect_t* expected = srNULL;
  srTestPointUnexpect_t unexpected[] = {
        {"X"},
        {"Y"},
        {"Z"},
        {0}};
...
}
</source>
'''Case 3.''' Need to verify that a subset of test points are not hit but everything else is expected.
Basically in that case a limited set of test points is unexpected:
{| class="prettytable"
| '''Expected'''
| '''Unexpected'''
| '''Ignored'''
|-
| "everything else"
| X, Z, Y
|
|}
<source lang="c">
#include <srtest.h>
void tf_testpoint_some(void)
{
  srTestPointExpect_t expected[] = {
        {srTEST_POINT_EVERYTHING_ELSE},
        {0}};
  srTestPointUnexpect_t unexpected[] = {
        {"X"},
        {"Y"},
        {"Z"},
        {0}};
...
}
</source>
=== System Observation ===
If want to collect all test points universally for purely investigation/diagnosis, not testing, then everything is expected, in other words:
{| class="prettytable"
| '''Expected'''
| '''Unexpected'''
| '''Ignored'''
|-
| "everything"
|
|
|}
To express that use the special srTEST_POINT_EVERYTHING_ELSE constant and do the following:
<source lang="c">
#include <srtest.h>
void tf_testpoint_none(void)
{
  srTestPointExpect_t expected[] = {
        {srTEST_POINT_EVERYTHING_ELSE},
        {0}};
  srTestPointUnexpect_t* unexpected = srNULL;
...
}
</source>
=== Wait Until Condition ===
If you need to ignore some test points until certain condition occurs and then proceed in a normal way:
{| class="prettytable"
| '''Expected'''
| '''Unexpected'''
| '''Ignored'''
|-
| "ignore"..., A, B, C, D
|
|
|}
To express that use the special srTEST_POINT_ANYTHING constant with associated predicate that returns <tt>srTRUE</tt> on met condition:
<source lang="c">
#include <srtest.h>
void tf_testpoint_none(void)
{
  srTestPointExpect_t expected[] = {
        {srTEST_POINT_ANYTHING, 1, MyPredicate},
        {"A"},
        {"B"},
        {"C"},
        {"D"},
        {0}};
  srTestPointUnexpect_t* unexpected = srNULL;
...
}
</source>
[[Category:Test Units]]

Latest revision as of 21:14, 8 July 2015

Test code to validate the Expectations of the Test Points often follows this basic pattern:

  1. Specify an expectation set consisting of expected (i.e. the test points that are expected to be hit) and optionally unexpected (i.e. the test points that are not expected to be hit) test points
  2. Register the expectation set with the Stride runtime
  3. Invoke the software under test (causing instrumentation points to be hit). This may not be necessary if the instrumented software under test is constantly running).
  4. Wait for the expectation set to be satisfied or a timeout to occur

Here is an example: 

#include <srtest.h>

void tf_testpoint_wait(void)
{
  /* specify expected set */
  srTestPointExpect_t expected[]= {
        {"START"}, 
        {"ACTIVE"}, 
        {"IDLE"},
        {"END"}, 
        {0}};

  /* specify unexpected set */
  srTestPointUnexpect_t unexpected[]= {
        {"INVALID"}, 
        {0}};

  /* register the expectation set*/
  srWORD handle;
  srTestPointSetup(expected, unexpected, srTEST_POINT_EXPECT_UNORDERED, srTEST_CASE_DEFAULT, &handle);

  /* start your asynchronous operation */
  ...

  /* wait for expectation set to be satisfied or a timeout to occur */
  srTestPointWait(handle, 1000);
}

#ifdef _SCL
#pragma scl_test_flist(“testfunc”, tf_testpoint_wait)
#endif


Reference

Expectation Set

An expectation set is specified with an expected array of srTestPointExpect_t structures and a second optional unexpected array of srTestPointUnexpect_t structures.

Expected Array

srTestPointExpect_t is typedef'd as follows: 

typedef struct
{
    /* the label value is considered the test point's identity */
    const srCHAR *          label;
    /* optional, count specifies the number of times the test point is expected to be hit */ 
    srDWORD                 count;
    /* optional, predicate function to use for payload validation against user data */ 
    srTestPointPredicate_t  predicate;
    /* optional, user data to validate the payload against */
    void *                  user;
} srTestPointExpect_t;

NOTES:

  • The end of the array has to be marked by a srTestPointExpect_t set to all zero values
  • The count, predicate and user members may be omitted in the array declaration (they will be automatically set to 0 by the compiler)
  • A count value of either 0 or 1 is interpreted as 1
  • The count could be set as "0 or more" by using the special srTEST_POINT_ANY_COUNT symbolic constant
  • A predicate value 0 indicates that any associated data with a test point payload will be ignored.
  • A user value 0 indicates that there is no user data associated with this test point
  • The label could be specified to any test point within the current expected set of test points by using the special srTEST_POINT_ANY_IN_SET or srTEST_POINT_ANY_AT_ALL symbolic constants.

Unexpected Array

srTestPointUnexpect_t is typedef'd as follows: 

typedef struct
{
    /* the label value is considered the test point's identity */
    const srCHAR *          label;
} srTestPointUnexpect_t;

NOTES:

  • The end of the array has to be marked by a srTestPointUnexpect_t set to all zero values
  • The label could be specified to everything else relative to the expected array by using the special srTEST_POINT_EVERYTHING_ELSE symbolic constant.

srTestPointPredicate_t

When defining the expectation set per entry a payload validation predicate function could be specified. The signature of it should match the following type: 

extern "C" typedef srBYTE (*srTestPointPredicate_t)(const srTestPoint_t* ptTP, void* pvUser);
Parameters Type Description
ptTP Input Pointer to the currently processed Test Point.
pvUser Input Pointer to opaque user data associated with an entry in the expectation set.


Return Value Description
srBYTE srTRUE for valid, srFALSE for invalid, srIGNORE otherwise.

NOTE:

  • As part of the standard Stride distribution there are three predefined function predicate helpers:
    • srTestPointMemCmp - byte comparison
    • srTestPointStrCmp - string case sensitive comparison
    • srTestPointStrCaseCmp - string case insensitive comparison

srTestPointSetup

The srTestPointSetup() routine is used to register an expectation set. 

srBOOL srTestPointSetup(srTestPointExpect_t* ptExpected, 
                        srTestPointUnexpect_t* ptUnexpected, 
                        srBYTE yMode, 
                        srTestCaseHandle_t tTestCase, 
                        srWORD* pwHandle);
Parameters Type Description
ptExpected Input Pointer to an expectated array.
ptUnexpected Input Pointer to an unexpectated array. This is optional and could be set srNULL.
yMode Input Bitmask that specifies whether the expectated test points occur in order and/or strict. Possible values are:

srTEST_POINT_EXPECT_ORDERED - the test points are expected to be hit exactly in the defined order
srTEST_POINT_EXPECT_UNORDERED - the test points could to be hit in any order
srTEST_POINT_EXPECT_STRICT - the test points are expected to be hit exactly as specified (no consecutive duplicate hits)
srTEST_POINT_EXPECT_NONSTRICT - other test points from the universe could to be hit in between
srTEST_POINT_EXPECT_CONTINUE - on successful expectation satisfaction continue processing until the wait timeout expires

tTestCase Input Handle to a test case. srTEST_CASE_DEFAULT can be used for the default test case.
pwHandle Output Handle that represents the registered expectation set


Return Value Description
srBOOL srTRUE on success, srFALSE otherwise.

srTestPointWait

The srTestPointWait() routine is used to wait for the expectation to be satisfied. 

srBOOL srTestPointWait(srWORD wHandle, 
                       srDWORD dwTimeout);
Parameters Type Description
wHandle Input Handle to a registered expectation set.
dwTimeout Input Timeout value in milliseconds; 0 means just check without waiting.


Return Value Description
srBOOL srTRUE on success, srFALSE otherwise.

NOTES:

  • The test thread blocks until either the expectation set is satisfied, unless srTEST_POINT_EXPECT_CONTINUE is specified on setup, or the timeout elapses.
  • All test points hit during the wait (both expected and unexpected) are added to the test report as testcase comments
  • Once the wait is over (whether the expectation set has been satisfied or there has been a test failure), the current expectation set is automatically unregistered from the Stride runtime and the handle is released
  • If you want to return immediately from a test case if expectation fails then make the check/wait call an argument to the srASSERT_TRUE() macro.

srTestPointCheck

The srTestPointCheck() routine is used to check for the expectation post routine completion. This is useful for verifying a set of expectations events that should have already transpired (thus are waiting to be processed). 

srBOOL srTestPointCheck(srWORD wHandle);
Parameters Type Description
wHandle Input Handle to a registered expectation set.


Return Value Description
srBOOL srTRUE on success, srFALSE otherwise.

NOTES:

  • All test points hit before the check (both expected and unexpected) are added to the test report as testcase comments
  • Once the check is done (whether the expectation set has been satisfied or there has been a test failure), the current expectation set is automatically unregistered from the Stride runtime and the handle is released
  • If you want to return immediately from a test case if expectation fails then make the check/wait call an argument to the srASSERT_TRUE() macro.

C++ Facade Class

The srtest.h file provides a simple facade class that wraps the test point APIs described above in a simple C++ class called srTestPointsHandler. If you are writing your unit tests in C++ (using Stride test classes), then this class is available for your use. The class implements the following methods, which correspond exactly to the C API equivalents:

srTestPointsHandler
constructor. Takes four arguments which match exactly the first four parameters of the srTestPointSetup function.
Wait
instance method that provides same functionality as srTestPointWait.
Check
instance method that provides same functionality as srTestPointCheck.
Retrieved from "https://www.stridewiki.com/index.php?title=Test_Point_Testing_in_C/C%2B%2B&oldid=14588"