Test Point Testing in C/C++

Testpoints provide an easy-to-use framework for solving a class of common yet difficult unit testing problems:

How can I observe and verify activity that occurs in another thread?

A couple of common scenarios that become a lot more testable via testpoints include:

• Verification of State machine operation
• Verification of communication drivers

Instrumenting Target Threads

Target threads are instrumented simply by placing lines of the following form into the source code:

...
/* a testpoint with no payload */
srTEST_POINT("first testpoint");

srTEST_POINT_DATA("second testpoint", "string payload");

srTEST_POINT_DATA1("third testpoint", "payload with format string %d", myVar);

When this code is executed it broadcasts a message via the STRIDE runtime which is detected by the test (IM) thread if it is currently looking for testpoints (i.e. in a srTEST_POINT_WAIT()). We refer to this as a testpoint hit.

Instrumenting the Test Thread

The test thread is instrumented using these steps:

1. Specify an expectation set (i.e. the testpoints that are expected to be hit)
2. Register the expectation set with the STRIDE runtime
3. Wait for the expectation set to be satisfied or a timeout to occur

Specifying the Expectation Set

An expectation set is an array of srTestPointExpect_t structures. srTestPointExpect_t is typdef'd as follows:

typedef struct
{
    /* the label value is considered the testpoint's identity */
    const srCHAR *  label;
    /* count specifies the number of times the testpoint is expected to be hit */ 
    srDWORD         count;
    /* data specifies an optional string data payload */
    const srCHAR *  data;
} srTestPointExpect_t;

Basic Example

A typical delcaration of an expectation set looks like this:

    srTestPointExpect_t expected[]= {
        {"START"}, 
        {"ACTIVE"}, 
        {"IDLE"},
        {"END"}, 
        {0}};

This example specifies the expectation of one hit each of the testpoints "START", "ACTIVE", "IDLE", and "END".

A few things to note:

• The end of the array is marked by a srTestPointExpect_t set to all zero values
• Structure members we omit in the array declaration are set to 0 by the compiler
• A count value of either 0 or 1 is interpreted as 1
• A data value 0 indicates that there is no payload data associated with this testpoint

Registering the Expectation Set

To register an expectation set, we call srTestPointExpect and receive a handle to identify the expectation set in the next step.

The registration looks like this:

srWORD handle;
srTestPointExpect(expected, &handle);

Once the expectation set is registered, any testpoint hits are buffered and will be available to the subsequent srTEST_POINT_WAIT(). If the activity we will be obvserving/verifying needs to be started (e.g. a state machine gets kicked off) this should be done after srTestPointExpect().

Waiting for the Expectation Set to be Satisfied

The final step is to wait for the expectation to be satisfied. We do this using the srTEST_POINT_WAIT() macro as shown below.

srTEST_POINT_WAIT(handle, expect_flags, timeout);

The macro takes three arguments:

srTEST_POINT_WAIT() arguments

Argument	Description
handle	The handle returned from the call to srTestPointExpect() used to register the expectation set
expect_flags	Value that customizes the expectation in terms of testpoint hit order and testpoint exclusivity (see below)
timeout	Timeout value in milliseconds; 0 means infinite timeout

The action the macro takes is as follows:

• The test thread (the thread macro is executed on) blocks until either the expectation set is satisfied or the timeout elpases. (Be careful with a timeout of 0, this can hang the test thread waiting for an expectation.)
• All testpoints hit during the wait (both expected and unexpected) are added to the test report as testcase comments
• If an unexpected testpoint is encountered (either out of order or not in the expectation set), or the timeout period elapses before the expectation set is satisfied the current test immediately fails
• Once the wait is over (whether the expectation set has been satisfied or there has been a test failure), the current expectation set is automatically unregistered from the STRIDE runtime and the handle is released

Expect Flags

The following table shows the effect on expectations using different expect_flag values.

expect_flags values

Expect Flags	Unexpected Testpoint causes test failure	Out of order Testpoint causes failure
0
srTEST_POINT_WAIT_ORDER		X
srTEST_POINT_WAIT_STRICT	X
srTEST_POINT_WAIT_ORDER | srTEST_POINT_WAIT_STRICT	X	X