Test Point Testing in C/C++

Introduction

Test Points provide an easy-to-use framework for solving a class of common yet difficult unit testing problems:

How can I observe and verify activity that occurs in another thread?

A couple of common scenarios that become a lot more testable via test points include:

• Verification of State machine operation
• Verification of asynchronous callbacks
• Verification of communication drivers

Instrumenting Source Code

Target threads are instrumented simply by placing lines of the following form into the source code:

...
/* a test point with no payload */
srTEST_POINT("first test point");

/* a test point with binary payload */
srTEST_POINT_DATA("second test point", myData, sizeofMyData);

/* a test point with string payload */
srTEST_POINT_STR1("third test point", "payload with format string %d", myVar);

When this code is executed it broadcasts a message via the STRIDE runtime which is detected by the test (IM) thread if it is currently looking for test points (i.e. in a srTEST_POINT_WAIT()). We refer to this as a test point hit.

Instrumenting Test Code

The test code is instrumented using these steps:

1. Specify an expectation set consisting of expected (i.e. the test points that are expected to be hit) and optionally unexpected (i.e. the test points that are not expected to be hit) test points
2. Register the expectation set with the STRIDE runtime
3. If the activity we will be obvserving/verifying needs to be started (e.g. a state machine gets kicked off) this should be done after here
4. Wait for the expectation set to be satisfied or a timeout to occur

Here is an example:

#include <srtest.h>

void tf_testpoint_wait(void)
{
  /* specify expected set */
  srTestPointExpect_t expected[]= {
        {"START"}, 
        {"ACTIVE"}, 
        {"IDLE"},
        {"END"}, 
        {0}};

  /* specify unexpected set */
  srTestPointUnexpect_t unexpected[]= {
        {"INVALID"}, 
        {0}};

  /* register the expectation set with the STRIDE */
  srWORD handle;
  srTestPointExpect(expected, unexpected, srTEST_POINT_EXPECT_UNORDERED, &handle);

  /* start your asynchronous operation */
  ...

  /* wait for expectation set to be satisfied or a timeout to occur */
  srTEST_POINT_WAIT(handle, 1000);
}

#ifdef _SCL
#pragma scl_test_flist(“testfunc”, tf_testpoint_wait)
#endif

Reference

Test Point

To specify a test point you should use one of the following macros:

Test Point
srTEST_POINT(label)	label is a pointer to a null-terminated string
srTEST_POINT_DATA(label, data, size)	label is a pointer to a null-terminated string data is a pointer to a byte sequence size is the size of the data in bytes
srTEST_POINT_STR(label, message)	label is a pointer to a null-terminated string message is a pointer to a null-terminated string
srTEST_POINT_STR[1..4](label, message, ...)	label is a pointer to a null-terminated string message is a pointer to a null-terminated format string ... variable list (up to 4) matching the format string

Expectation Set

An expectation set is specified with an array of srTestPointExpect_t structures and a second optional array of srTestPointUnexpect_t structures.

Expected Array

srTestPointExpect_t is typedef'd as follows:

typedef struct
{
    /* the label value is considered the test point's identity */
    const srCHAR *          label;
    /* optional, count specifies the number of times the test point is expected to be hit */ 
    srDWORD                 count;
    /* optional, predicate function to use for payload validation against user data */ 
    srTestPointPredicate_t  predicate;
    /* optional, user data to validate the payload against */
    void *                  user;
} srTestPointExpect_t;

NOTES:

• The end of the array has to be marked by a srTestPointExpect_t set to all zero values
• The count, predicate and user members may be omitted in the array declaration (they will be automatically set to 0 by the compiler)
• A count value of either 0 or 1 is interpreted as 1
• The count could be set as "0 or more" by using the special srTEST_POINT_ANY_COUNT symbolic constant
• A predicate value 0 indicates that any associated with a test point payload will be ignored.
• A user value 0 indicates that there is no user data associated with this test point
• The label could be specified to everything else relative to the unexpected array by using the special symbolic constant srTEST_POINT_EVERYTHING_ELSE. When used it is required to be the one end only (non zero) entry in the array.

Unexpected Array

srTestPointUnexpect_t is typedef'd as follows:

typedef struct
{
    /* the label value is considered the test point's identity */
    const srCHAR *          label;
} srTestPointUnexpect_t;

NOTES:

• The end of the array has to be marked by a srTestPointUnexpect_t set to all zero values
• The label could be specified to everything else relative to the expected array by using the special symbolic constant srTEST_POINT_EVERYTHING_ELSE. When used it is required to be the one end only (non zero) entry in the array.

srTestPointPredicate_t

When defining the expectation set per entry a payload validation predicate function could be specified. The signature of it should match the following type:

typedef srBYTE (*srTestPointPredicate_t)(const srTestPoint_t* ptTP, void* pvUser);

Parameters	Type	Description
ptTP	Input	Pointer to the currently processed Test Point.
pvUser	Input	Pointer to opaque user data associated with an entry in the expectation set.

Return Value	Description
srBYTE	srTRUE on success, srFALSE on failure, srIGNORE otherwise.

NOTE:

• As part of the standard STRIDE distribution there are three predefined function predicate helpers:
  • srTestPointMemCmp - byte comparison
  • srTestPointStrCmp - string case sensitive comparison
  • srTestPointStrCaseCmp - string case insensitive comparison

srTestPointExpect

The srTestPointExpect() routine is used to register an expectation set.

srBOOL srTestPointExpect(srTestPointExpect_t* ptExpected, 
                         srTestPointUnexpect_t* ptUnexpected, 
                         srTestPointExpectOrder_e eOrder, 
                         srWORD* pwHandle);

Parameters	Type	Description
ptExpected	Input	Pointer to an expectated array.
ptUnexpected	Input	Pointer to an unexpectated array. This is optional and could be set srNULL.
eOrder	Input	Expectation order. Possible values are: srTEST_POINT_EXPECT_ORDERED - the test points are expected to be hit exactly in the defined order srTEST_POINT_EXPECT_UNORDERED - the test points could to be hit in any order
pwHandle	Output	Handle that represents the registered expectation set

Return Value	Description
srBOOL	srTRUE on success, srFALSE otherwise.

Expectation Validation

The behavior of the test point processing is as follows:

• If an unexpected test point is seen under test point wait, the expectation is failed and the wait is abandoned immediately.
• If an expected test point is seen under test point wait, the behavior varies depending on whether the expectation with the matching label includes a predicate function.
  • Without Predicate, the expectation is unconditionally considered “hit,” and the expected count is decremented. If the expected count reaches zero, the expectation is considered satisfied. (In other words, the behavior is identical to an expectation with a predicate where the predicate returns srTRUE.)
  • With Predicate, the predicate function determines the behavior depending on its return value. The predicate function is passed a copy of the data supplied with the original test point hit (along with other information) with which to determine its desired behavior. Depending on the return value:
    • srTRUE – the corresponding expectation is considered satisfied
    • srFALSE – the test point wait is abandoned and test case associated with the wait is set to fail status
    • srIGNORE – the test point hit is ignored
• If the expectation is not satisfied for the specified timeout period the expectation is considered failed. This rule has several exceptions:
  • Negative Expectations, don't expect some test points to be hit.
  • System Observation, want to collect all test points.

srTestPointWait

The srTestPointWait() routine is used to wait for the expectation to be satisfied.

srBOOL srTestPointWait(srWORD wHandle, 
                       srTestCaseHandle_t tTestCase, 
                       srDWORD dwTimeout);

Parameters	Type	Description
wHandle	Input	Handle to a registered expectation set.
tTestCase	Input	Handle to a test case where the results would be reported. srTEST_CASE_DEFAULT can be used for the default test case.
dwTimeout	Input	Timeout value in milliseconds; 0 means just check without waiting.

Return Value	Description
srBOOL	srTRUE on success, srFALSE otherwise.

For convinience the following macros are provided:

#define srTEST_POINT_WAIT(handle, timeout) srTestPointWait(handle, srTEST_CASE_DEFAULT, timeout)
#define srTEST_POINT_CHECK(handle) srTestPointWait(handle, srTEST_CASE_DEFAULT, 0)

NOTES:

• The test thread blocks until either the expectation set is satisfied or the timeout elapses.
• All test points hit during the wait (both expected and unexpected) are added to the test report as testcase comments
• If an unexpected test point is encountered (either out of order or not in the expectation set), or the timeout period elapses before the expectation set is satisfied the current test immediately fails
• Once the wait is over (whether the expectation set has been satisfied or there has been a test failure), the current expectation set is automatically unregistered from the STRIDE runtime and the handle is released
• if you want to return immediately from a test case if expectation fails then make the check/wait call an argument to srASSERT_TRUE().

Use Cases

Known Expectations

In reality the amount of Test Points in an application under test is mutch larger then the number of test points of interest. Basically, the known expectation set is limited and consists of {A, B, C, D, X, Y, Z}, where some are expected and other unexpected, and everything else is unknown and should be ignored. In other words:

Expected	Unexpected	Ignored
A, B, C, D	X, Y, Z	"everything else"

To express that do the following:

#include <srtest.h>
 
void tf_testpoint_known(void)
{
  srTestPointExpect_t expected[]= {
        {"A"}, 
        {"B"}, 
        {"C"}, 
        {"D"}, 
        {0}};

  srTestPointUnexpect_t unexpected[]= {
        {"X"}, 
        {"Y"}, 
        {"Z"}, 
        {0}};
...
}

Full Expectations

If the whole application under test is known, the expectation set consists of all test points, where a limited set is expected and consists of {A, B, C, D} and the rest are unexpected, then nothing should be ignored. In other words:

Expected	Unexpected	Ignored
A, B, C, D	"everything else"

To express that use the special srTEST_POINT_EVERYTHING_ELSE constant and do the following:

#include <srtest.h>
 
void tf_testpoint_all(void)
{
  srTestPointExpect_t expected[]= {
        {"A"}, 
        {"B"}, 
        {"C"}, 
        {"D"}, 
        {0}};

  srTestPointUnexpect_t unexpected[]= {
        {srTEST_POINT_EVERYTHING_ELSE}, 
        {0}};
...
}

0 or More Expectations

In some cases the expected test point pattern is something like:

• START
• PROGRESS

...

• END

where any number (0 or more) of PROGRESS are expected but doesn't matter how many.

To specify that use the special srTEST_POINT_ANY_COUNT constant:

#include <srtest.h>
 
void tf_testpoint_any(void)
{
  srTestPointExpect_t expected[]= {
        {"START"}, 
        {"PROGRESS", srTEST_POINT_ANY_COUNT}, 
        {"END"}, 
        {0}};
...
}

Payload Expectations

In some cases the expected test point may carry a payload that needs to be validated against a user defined data:

• START
• PROGRESS_BIN({1,2,3})
• PROGRESS_STR("abc")
• END

where PROGRESS_BIN is expected to carry a binary payload and PROGRESS_STR - string payload.

To express that specify a predicate and user defined data:

#include <srtest.h>
 
void tf_testpoint_payload(void)
{
  srBYTE pyData[] = {1, 2, 3};
  srTestPointExpect_t expected[]= {
        {"START"}, 
        {"PROGRESS_BIN", srTestPointMemCmp, pyData}, 
        {"PROGRESS_STR", srTestPointStrCmp, "abc"}, 
        {"END"}, 
        {0}};
...
}

Negative Expectations

Case 1. Need to run a scenario and verify that NONE of the test points are hit.

Basically in that case everything is unexpected:

Expected	Unexpected	Ignored
	"everything"

To express that use the special srTEST_POINT_EVERYTHING_ELSE constant and do the following:

#include <srtest.h>
 
void tf_testpoint_none(void)
{
  srTestPointExpect_t* pExpected = srNULL;

  srTestPointUnexpect_t unexpected[] = {
        {srTEST_POINT_EVERYTHING_ELSE},
        {0}};

...
}

Case 2. Need to verify that a subset of test points are not hit and everything else should be ignored.

Basically in that case a limited set of test points is unexpected:

Expected	Unexpected	Ignored
	X, Z, Y	"everything else"

#include <srtest.h>
 
void tf_testpoint_some(void)
{
  srTestPointExpect_t* pExpected = srNULL;

  srTestPointUnexpect_t unexpected[] = {
        {"X"},
        {"Y"},
        {"Z"},
        {0}};
...
}

Case 3. Need to verify that a subset of test points are not hit but everything else is expected.

Basically in that case a limited set of test points is unexpected:

Expected	Unexpected	Ignored
"everything else"	X, Z, Y

#include <srtest.h>
 
void tf_testpoint_some(void)
{
  srTestPointExpect_t expected[] = {
        {srTEST_POINT_EVERYTHING_ELSE},
        {0}};

  srTestPointUnexpect_t unexpected[] = {
        {"X"},
        {"Y"},
        {"Z"},
        {0}};
...
}

System Observation

If want to collect all test points universally for purely investigation/diagnosis, not testing, then everything is expected, In other words:

Expected	Unexpected	Ignored
"everything"

To express that use the special srTEST_POINT_EVERYTHING_ELSE constant and do the following:

#include <srtest.h>
 
void tf_testpoint_none(void)
{
  srTestPointExpect_t expected[] = {
        {srTEST_POINT_EVERYTHING_ELSE},
        {0}};

  srTestPointUnexpect_t unexpected = srNULL;

...
}