Test Point Testing in C/C++
Introduction
Test Points provide an easy-to-use framework for solving a class of common yet difficult unit testing problems:
How can I observe and verify activity that occurs in another thread?
A couple of common scenarios that become a lot more testable via test points include:
- Verification of State machine operation
- Verification of communication drivers
Instrumenting Target Code
Target threads are instrumented simply by placing lines of the following form into the source code:
...
/* a test point with no payload */
srTEST_POINT("first test point");
srTEST_POINT_DATA("second test point", "string payload");
srTEST_POINT_DATA1("third test point", "payload with format string %d", myVar);
When this code is executed it broadcasts a message via the STRIDE runtime which is detected by the test (IM) thread if it is currently looking for test points (i.e. in a srTEST_POINT_WAIT()). We refer to this as a test point hit.
Instrumenting the Test Code
The test code is instrumented using these steps:
- Specify an expectation set (i.e. the test points that are expected to be hit)
- Register the expectation set with the STRIDE runtime
- Wait for the expectation set to be satisfied or a timeout to occur
Here is an example:
#include <srtest.h>
void tf_testpoint_wait(void)
{
/* specify expectation set */
srTestPointExpect_t expected[]= {
{"START"},
{"ACTIVE"},
{"IDLE"},
{"END"},
{0}};
/* register the expectation set with the STRIDE */
srWORD handle;
srTestPointExpect(expected, srTEST_POINT_EXPECT_UNORDERED, srTEST_POINT_EXPECT_NONEXCLUSIVE, &handle);
/* start your asynchronous operation */
...
/* wait for expectaion set to be satisfied or a timeout to occur */
srTEST_POINT_WAIT(handle, 1000);
}
#ifdef _SCL
#pragma scl_test_flist(“testfunc”, tf_testpoint_wait)
#endif
Reference
Expectation Set
An expectation set is an array of srTestPointExpect_t structures. srTestPointExpect_t is typdef'd as follows:
typedef struct
{
/* the label value is considered the test point's identity */
const srCHAR * label;
/* count specifies the number of times the test point is expected to be hit */
srDWORD count;
/* data specifies an optional string data payload */
const srCHAR * data;
} srTestPointExpect_t;
A few things to note:
- The end of the array has to be marked by a srTestPointExpect_t set to all zero values
- The count and data members could be omit in the array declaration (they will be automatically set to 0 by the compiler)
- A count value of either 0 or 1 is interpreted as 1
- A data value 0 indicates that there is no payload data associated with this test point
srTestPointExpect
The srTestPointExpect() routine is used to register an expecttaion set.
srBOOL srTestPointExpect(srTestPointExpect_t* ptExpected, srTestPointExpectOrder_e eOrder, srTestPointExpectMembership_e eMembership, srWORD* pwHandle)
Parameters | Type | Description |
ptExpected | Input | Pointer to an expectaion set. |
eOrder | Input | Expectation order. Posible values are: srTEST_POINT_EXPECT_ORDERED - the test points are expected to be hit exactly in the defined order |
eMembership | Input | Expectation membership. Posible values are: srTEST_POINT_EXPECT_EXCLUSIVE - any non expected test point hit will be treated as an error |
Return Value | Description |
srOK | Success |
srERR | Null string passed in. |
srTEST_ERR_HANDLE_INVALID | Invalid handle passed in. |
srTEST_WARN_STR_TRUNCATED | String passed in was too large and was truncated. |
This example specifies that the order of expectations is not important (srTEST_POINT_EXPECT_UNORDERED) and the membership of expectations is not exclusive (srTEST_POINT_EXPECT_NONEXCLUSIVE).
Once the expectation set is registered, any test point hits are buffered and will be available to the subsequent srTEST_POINT_WAIT(). If the activity we will be obvserving/verifying needs to be started (e.g. a state machine gets kicked off) this should be done after srTestPointExpect().
Waiting for the Expectation Set to be Satisfied
The final step is to wait for the expectation to be satisfied. We do this using the srTEST_POINT_WAIT() macro as shown below.
srTEST_POINT_WAIT(handle, expect_flags, timeout);
The macro takes three arguments:
Argument | Description |
handle | The handle returned from the call to srTestPointExpect() used to register the expectation set |
expect_flags | Value that customizes the expectation in terms of test point hit order and test point exclusivity (see below) |
timeout | Timeout value in milliseconds; 0 means infinite timeout |
The action the macro takes is as follows:
- The test thread (the thread macro is executed on) blocks until either the expectation set is satisfied or the timeout elpases. (Be careful with a timeout of 0, this can hang the test thread waiting for an expectation.)
- All test points hit during the wait (both expected and unexpected) are added to the test report as testcase comments
- If an unexpected test point is encountered (either out of order or not in the expectation set), or the timeout period elapses before the expectation set is satisfied the current test immediately fails
- Once the wait is over (whether the expectation set has been satisfied or there has been a test failure), the current expectation set is automatically unregistered from the STRIDE runtime and the handle is released
Expect Flags
The following table shows the effect on expectations using different expect_flag values.
Expect Flags | Unexpected Test Point causes test failure |
Out of order Test Point causes failure |
0 | ||
srTEST_POINT_WAIT_ORDER | X | |
srTEST_POINT_WAIT_STRICT | X | |
srTEST_POINT_WAIT_ORDER | srTEST_POINT_WAIT_STRICT | X | X |