View source for Test Point Testing in C/C++

== Introduction ==
Test Points provide an easy-to-use framework for solving a class of common yet difficult unit testing problems: 

 ''How can I observe and verify activity that occurs in another thread?''

A couple of common scenarios that become a lot more testable via test points include:
* Verification of State machine operation
* Verification of asynchronous callbacks
* Verification of communication drivers

== Instrumenting Source Code ==
Target threads are instrumented simply by placing lines of the following form into the source code:
<source lang='c'>
...
/* a test point with no payload */
srTEST_POINT("first test point");

srTEST_POINT_DATA("second test point", "string payload");

srTEST_POINT_DATA1("third test point", "payload with format string %d", myVar);
</source>

When this code is executed it broadcasts a message via the STRIDE runtime which is detected by the test (IM) thread if it is currently looking for test points (i.e. in a srTEST_POINT_WAIT()). We refer to this as a ''test point hit''.

== Instrumenting Test Code ==
The test code is instrumented using these steps:

# Specify an expectation set (i.e. the test points that are expected to be hit)
# Register the expectation set with the STRIDE runtime
# If the activity we will be obvserving/verifying needs to be started (e.g. a state machine gets kicked off) this should be done after here
# Wait for the expectation set to be satisfied or a timeout to occur

Here is an example:

<source lang="c">
#include <srtest.h>

void tf_testpoint_wait(void)
{
  /* specify expectation set */
  srTestPointExpect_t expected[]= {
        {"START"}, 
        {"ACTIVE"}, 
        {"IDLE"},
        {"END"}, 
        {0}};

  /* register the expectation set with the STRIDE */
  srWORD handle;
  srTestPointExpect(expected, srTEST_POINT_EXPECT_UNORDERED, srTEST_POINT_EXPECT_NONEXCLUSIVE, &handle);

  /* start your asynchronous operation */
  ...

  /* wait for expectation set to be satisfied or a timeout to occur */
  srTEST_POINT_WAIT(handle, 1000);
}

#ifdef _SCL
#pragma scl_test_flist(“testfunc”, tf_testpoint_wait)
#endif
</source>

== Reference ==

=== Test Point ===
To specify a test point you should use one of the following macros:

{| class="prettytable"
| colspan="2" | '''Test Point'''
|-
| srTEST_POINT(''label'')
| ''label'' is a pointer to a null-terminated string
|-
| srTEST_POINT_DATA(''label'', ''message'')
| ''label'' is a pointer to a null-terminated string<br/>
''message'' is a pointer to a null-terminated string
|-
| srTEST_POINT_DATA[1..4](''label'', ''message'', ...)
| ''label'' is a pointer to a null-terminated string<br/>
''message'' is a pointer to a null-terminated format string<br/>
''...'' variable list (up to 4) matching the format string
|}

=== Expectation Set ===
An expectation set is an array of '''srTestPointExpect_t''' structures. srTestPointExpect_t is typedef'd as follows:

<source lang='c'>
typedef struct
{
    /* the label value is considered the test point's identity */
    const srCHAR *  label;
    /* count specifies the number of times the test point is expected to be hit */ 
    srDWORD         count;
    /* data specifies an optional string data payload */
    const srCHAR *  data;
} srTestPointExpect_t;
</source>

'''NOTES:'''
* The end of the array has to be marked by a srTestPointExpect_t set to all zero values
* The ''count'' and ''data'' members may be omitted in the array declaration (they will be automatically set to 0 by the compiler)
* A ''count'' value of either 0 or 1 is interpreted as 1 
* The ''count'' could be set as "0 or more" by using the special srTEST_POINT_ANY_COUNT symbolic constant  
* A ''data'' value 0 indicates that there is no payload data associated with this test point

=== srTestPointExpect ===
The srTestPointExpect() routine is used to register an expectation set.

<source lang="c">
srBOOL srTestPointExpect(srTestPointExpect_t* ptExpected, 
                         srTestPointExpectOrder_e eOrder, 
                         srTestPointExpectMembership_e eMembership, 
                         srWORD* pwHandle);
</source>

{| border="1" cellspacing="0" cellpadding="10" style="align:left;"  
| '''Parameters''' 
| '''Type'''
| '''Description'''
|-
| ptExpected
| Input
| Pointer to an expectation set.
|-
| eOrder 
| Input 
| Expectation order. Possible values are: <br/> 
srTEST_POINT_EXPECT_ORDERED - the test points are expected to be hit exactly in the defined order <br/>
srTEST_POINT_EXPECT_UNORDERED - the test points could to be hit in any order
|-
| eMembership 
| Input 
| Expectation membership. Possible values are: <br/> 
srTEST_POINT_EXPECT_EXCLUSIVE - any non expected test point hit will be treated as an error <br/>
srTEST_POINT_EXPECT_UNORDERED - non expected test points hit are ignored
|-
| pwHandle 
| Output 
| Handle that represents the registered expectation set
|}
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;"
| '''Return Value'''
| ''' Description'''
|-
| srBOOL 
| srTRUE on success, srFALSE otherwise.
|}

=== srTestPointWait ===
The srTestPointWait() routine is used to wait for the expectation to be satisfied. 

<source lang="c">
srBOOL srTestPointWait(srWORD wHandle, 
                       srTestCaseHandle_t tTestCase, 
                       srDWORD dwTimeout);
</source>

{| border="1" cellspacing="0" cellpadding="10" style="align:left;"  
| '''Parameters''' 
| '''Type'''
| '''Description'''
|-
| wHandle 
| Input
| Handle to a registered expectation set.
|-
| tTestCase 
| Input 
| Handle to a test case where the results would be reported. srTEST_CASE_DEFAULT can be used for the default test case.
|-
| dwTimeout 
| Input 
| Timeout value in milliseconds; 0 means just check without waiting.
|}
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;"
| '''Return Value'''
| ''' Description'''
|-
| srBOOL 
| srTRUE on success, srFALSE otherwise.
|}

For convinience the following macros are provided:
<source lang="c">
#define srTEST_POINT_WAIT(handle, timeout) srTestPointWait(handle, srTEST_CASE_DEFAULT, timeout)
#define srTEST_POINT_CHECK(handle) srTestPointWait(handle, srTEST_CASE_DEFAULT, 0)
</source>

NOTES:
* The test thread blocks until either the expectation set is satisfied or the timeout elapses.
* All test points hit during the wait (both expected and unexpected) are added to the test report as testcase comments
* If an unexpected test point is encountered (either out of order or not in the expectation set), or the timeout period elapses before the expectation set is satisfied the current test immediately fails
* Once the wait is over (whether the expectation set has been satisfied or there has been a test failure), the current expectation set is automatically unregistered from the STRIDE runtime and the handle is released

== Use Cases ==

=== 0 or More Expectations ===
In some cases the expected test point pattern is something like:
* START
* PROGRESS
...
* END
where any number (0 or more) of PROGRESS are expected but doesn't matter how many.

To specify that use the special <tt>srTEST_POINT_ANY_COUNT</tt> constant:
<source lang="c">
#include <srtest.h>
 
void tf_testpoint_any(void)
{
  /* specify expectation set */
  srTestPointExpect_t expected[]= {
        {"START"}, 
        {"PROGRESS", srTEST_POINT_ANY_COUNT}, 
        {"END"}, 
        {0}};
...
}
</source>

=== Negative Expectations === 
'''Case 1.''' Need to run a scenario and verify that NONE of the test points are hit.

Dedicate a special test point, lets call it “NEGATIVE_FINAL”, and add it at the end of your test sequence then by setting the expectation “exclusively” to just that test point would allow testing that none of your other test points were hit:

<source lang="c">
#include <srtest.h>
 
void tf_testpoint_none(void)
{
  srTestPointExpect_t expected[]= {
        {"NEGATIVE_FINAL"}, 
        {0}};

  srWORD handle;
  srTestPointExpect(expected, srTEST_POINT_EXPECT_UNORDERED, srTEST_POINT_EXPECT_EXCLUSIVE, &handle);
...
}
</source>


'''Case 2.''' Need to verify that a subset of test points are not hit.

By expecting that all other (not in the negative subset) test points could be hit any number of times combined with the the technique described in case 1 would allow testing that a subset was to hit.

<source lang="c">
#include <srtest.h>
 
void tf_testpoint_none(void)
{
  srTestPointExpect_t expected[]= {
        {"TP_1", srTEST_POINT_ANY_COUNT}, 
        {"TP_2", srTEST_POINT_ANY_COUNT}, 
        ...
        {"TP_N", srTEST_POINT_ANY_COUNT}, 
        {"NEGATIVE_FINAL"}, 
        {0}};

  srWORD handle;
  srTestPointExpect(expected, srTEST_POINT_EXPECT_UNORDERED, srTEST_POINT_EXPECT_EXCLUSIVE, &handle);
...
}
</source>


'''Case 3.''' Want to return immediately from a test case if expectation fails.

Note that <tt>[[#srTestPointWait|srTestPointWait()]]</tt> returns <tt>srFALSE</tt> if the expectation fails. By making the call an argument to <tt>[[Pass/Fail_Macros#Boolean_Macros|srASSERT_TRUE()]]</tt>, you can automatically return from the current test case if the expectation is not met.

<source lang="c">

...

/* If expectation fails, returns immediately */
srASSERT_TRUE(srTestPointWait(handle, 1000));

</source>

[[Category:Test Units]]
[[Category:Testing]]