Test Point Testing in C/C++

Introduction

Test Points provide an easy-to-use framework for solving a class of common yet difficult unit testing problems:

How can I observe and verify activity that occurs in another thread?

A couple of common scenarios that become a lot more testable via test points include:

• Verification of State machine operation
• Verification of communication drivers

Instrumenting Target Code

Target threads are instrumented simply by placing lines of the following form into the source code:

...
/* a test point with no payload */
srTEST_POINT("first test point");

srTEST_POINT_DATA("second test point", "string payload");

srTEST_POINT_DATA1("third test point", "payload with format string %d", myVar);

When this code is executed it broadcasts a message via the STRIDE runtime which is detected by the test (IM) thread if it is currently looking for test points (i.e. in a srTEST_POINT_WAIT()). We refer to this as a test point hit.

Instrumenting the Test Code

The test code is instrumented using these steps:

1. Specify an expectation set (i.e. the test points that are expected to be hit)
2. Register the expectation set with the STRIDE runtime
3. Wait for the expectation set to be satisfied or a timeout to occur

Here is an example:

#include <srtest.h>

void tf_testpoint_wait(void)
{
  /* specify expectation set */
  srTestPointExpect_t expected[]= {
        {"START"}, 
        {"ACTIVE"}, 
        {"IDLE"},
        {"END"}, 
        {0}};

  /* register the expectation set with the STRIDE */
  srWORD handle;
  srTestPointExpect(expected, srTEST_POINT_EXPECT_UNORDERED, srTEST_POINT_EXPECT_NONEXCLUSIVE, &handle);

  /* start your asynchronous operation */
  ...

  /* wait for expectaion set to be satisfied or a timeout to occur */
  srTEST_POINT_WAIT(handle, 1000);
}

#ifdef _SCL
#pragma scl_test_flist(“testfunc”, tf_testpoint_wait)
#endif

Reference

Expectation Set

An expectation set is an array of srTestPointExpect_t structures. srTestPointExpect_t is typdef'd as follows:

typedef struct
{
    /* the label value is considered the test point's identity */
    const srCHAR *  label;
    /* count specifies the number of times the test point is expected to be hit */ 
    srDWORD         count;
    /* data specifies an optional string data payload */
    const srCHAR *  data;
} srTestPointExpect_t;

A few things to note:

• The end of the array has to be marked by a srTestPointExpect_t set to all zero values
• The count and data members could be omit in the array declaration (they will be automatically set to 0 by the compiler)
• A count value of either 0 or 1 is interpreted as 1
• A data value 0 indicates that there is no payload data associated with this test point

srTestPointExpect

The srTestPointExpect() routine is used to register an expecttaion set.

srBOOL srTestPointExpect(srTestPointExpect_t* ptExpected, srTestPointExpectOrder_e eOrder, srTestPointExpectMembership_e eMembership, srWORD* pwHandle)

Parameters	Type	Description
ptExpected	Input	Pointer to an expectaion set.
eOrder	Input	Expectation order. Posible values are: srTEST_POINT_EXPECT_ORDERED - the test points are expected to be hit exactly in the defined order srTEST_POINT_EXPECT_UNORDERED - the test points could to be hit in any order
eMembership	Input	Expectation membership. Posible values are: srTEST_POINT_EXPECT_EXCLUSIVE - any non expected test point hit will be treated as an error srTEST_POINT_EXPECT_UNORDERED - non expected test points hit are ignored

Return Value	Description
srOK	Success
srERR	Null string passed in.
srTEST_ERR_HANDLE_INVALID	Invalid handle passed in.
srTEST_WARN_STR_TRUNCATED	String passed in was too large and was truncated.

This example specifies that the order of expectations is not important (srTEST_POINT_EXPECT_UNORDERED) and the membership of expectations is not exclusive (srTEST_POINT_EXPECT_NONEXCLUSIVE).

Once the expectation set is registered, any test point hits are buffered and will be available to the subsequent srTEST_POINT_WAIT(). If the activity we will be obvserving/verifying needs to be started (e.g. a state machine gets kicked off) this should be done after srTestPointExpect().

Waiting for the Expectation Set to be Satisfied

The final step is to wait for the expectation to be satisfied. We do this using the srTEST_POINT_WAIT() macro as shown below.

srTEST_POINT_WAIT(handle, expect_flags, timeout);

The macro takes three arguments:

srTEST_POINT_WAIT() arguments

Argument	Description
handle	The handle returned from the call to srTestPointExpect() used to register the expectation set
expect_flags	Value that customizes the expectation in terms of test point hit order and test point exclusivity (see below)
timeout	Timeout value in milliseconds; 0 means infinite timeout

The action the macro takes is as follows:

• The test thread (the thread macro is executed on) blocks until either the expectation set is satisfied or the timeout elpases. (Be careful with a timeout of 0, this can hang the test thread waiting for an expectation.)
• All test points hit during the wait (both expected and unexpected) are added to the test report as testcase comments
• If an unexpected test point is encountered (either out of order or not in the expectation set), or the timeout period elapses before the expectation set is satisfied the current test immediately fails
• Once the wait is over (whether the expectation set has been satisfied or there has been a test failure), the current expectation set is automatically unregistered from the STRIDE runtime and the handle is released

Expect Flags

The following table shows the effect on expectations using different expect_flag values.

Expect Flags	Unexpected Test Point causes test failure	Out of order Test Point causes failure
0
srTEST_POINT_WAIT_ORDER		X
srTEST_POINT_WAIT_STRICT	X
srTEST_POINT_WAIT_ORDER | srTEST_POINT_WAIT_STRICT	X	X