Indicating Pass/Fail in a Test Method

Test Method Return Values

As described elsewhere, Stride test methods must conform to certain criteria. Among other things, these criteria specify allowable signatures for test methods.

Three distinct return return types are permitted, each with their own behavior with respect to how return values are interpreted by the Stride runtime in setting PASS/FAIL status.

Following is a table summarizing the details of this behavior and code examples illustrating the use of each return type.

Return Type	Description	How Return Value is Interpreted	Default Status
void	Most common return type	Since there is no return, PASS/FAIL status is set in the body of the method using a test macro or a runtime call srTestCaseSetStatus() srTestCaseSetStatusEx() srTestCase::SetStatus()	If the method runs to completion without another status being set, the status is srTEST_PASS If the method does not run to completion (due to crash or hang in software under test), the status is srTEST_INPROGRESS
integer type	Integer types include: int short char long may include signed/unsigned/const qualifiers	If you explicitly set the status in the method (using a test macro or a runtime call), this status will override the status that would otherwise be set by the return value If you don't explicitly set the status in the method, the return value determines the status as follows: Return is zero -> srTEST_PASS Return is non-zero -> srTEST_FAIL	If the method runs to completion, the status is determined by the rules to the left (there is no default) If the method does not run to completion (due to crash or hang in software under test), the status is srTEST_INPROGRESS
bool	c++ only	If you explicitly set the status in the method (using a test macro or a runtime call), this status will override the status that would otherwise be set by the return value If you don't explicitly set the status in the method, the return value determines the status as follows: Return is true -> srTEST_PASS Return is false -> srTEST_FAIL	If the method runs to completion, the status is determined by the rules to the left (there is no default) If the method does not run to completion (due to crash or hang in software under test), the status is srTEST_INPROGRESS

Examples

The following example implementations validate the operation of a function named DoAdd() (not shown). Required header file declarations are assumed to be in place.

c++ examples are shown, but techniques are applicable to c as well.

void return

This is the return type that is most often used as it provides the most convenience.

#include <srtest.h>

/////////////////////////////
// shows use of a test macro to set status
/////////////////////////////
void MyTest::Test1()
{
    int sum = DoAdd(2, 2);
    // test macro validates condition and sets test status.
    srEXPECT_EQ(sum, 4);
}

/////////////////////////////
// shows use a a runtime call to set status
/////////////////////////////
void MyTest::Test2()
{
    int sum = DoAdd(3, 3);
    // set the status using a runtime call; much less convenient than
    // using the macro, but same result
    srTestStatus_e eStatus = (sum == 6) ? srTEST_PASS : srTEST_FAIL;
    testCase.SetStatus(eStatus, 0);
}

/////////////////////////////
// shows use of ASSERT macro to "short circuit" the test
/////////////////////////////
void MyTest::Test3()
{
    HANDLE h = GetResource();
    // if the resource is not acquired, there is no point in continuing; the
    // ASSERT macro validates and will set status to srTEST_FAIL and 
    // immediately return if the condition is not met
    srASSERT_NE(h, 0);

    int rc = h->DoSomething();
    srEXPECT_EQ(rc, 0);
}

/////////////////////////////
// shows default status
/////////////////////////////
void MyTest::Test4()
{
    // status is not set, so default srTEST_PASS is in force
}

integer return

#include <srtest.h>

/////////////////////////////
// shows use of return value to set status
/////////////////////////////
int MyTest::Test1()
{
    int sum = DoAdd(2, 2);
    // return 0 -> srTEST_PASS, non-zero -> srTEST_FAIL
    return (sum == 4) ? 0 : 1;
}

/////////////////////////////
// shows use of test macro to override return value
/////////////////////////////
int MyTest::Test2()
{
    int sum = DoAdd(3, 3);
    srEXPECT_EQ(sum, 6);
    // return value does not affect test status since the status
    // was set explicitly using a test macro above
    return 0;
}

/////////////////////////////
// shows use of ASSERT macro in test with non-void return type
/////////////////////////////
int MyTest::Test3()
{
    HANDLE h = GetResource();
    // if the resource is not acquired, there is no point in continuing; the
    // ASSERT macro validates and will set status to srTEST_FAIL and 
    // immediately return if the condition is not met
    //
    // Note the unusual syntax here; this allows the ASSERT macro (usually used where
    // void return type is in force) to be used here. If the ASSERT condition is not met,
    // the status is set to srTEST_FAIL and the method returns the value to the right of
    // the comma. The actual value is immaterial since by setting the status explicitly
    // we override the normal meaning of the return value.
    srASSERT_NE(h, 0), -1;

    int rc = h->DoSomething();
    srEXPECT_EQ(rc, 0);

    return 0;
}

bool return

#include <srtest.h>

bool MyTest::Test1()
{
    int sum = DoAdd(2, 2);
    // return true -> PASS, false -> FAIL
    return (sum == 4);
}

bool MyTest::Test2()
{
    int sum = DoAdd(3, 3);
    // 
    srEXPECT_EQ(sum, 6);
    // return value does not affect test status since the status
    // was set explicitly using a test macro above
    return true;
}

/////////////////////////////
// shows use of ASSERT macro in test with non-void return type
/////////////////////////////
bool MyTest::Test3()
{
    HANDLE h = GetResource();
    // if the resource is not acquired, there is no point in continuing; the
    // ASSERT macro validates and will set status to srTEST_FAIL and 
    // immediately return if the condition is not met
    //
    // Note the unusual syntax here; this allows the ASSERT macro (usually used where
    // void return type is in force) to be used here. If the ASSERT condition is not met,
    // the status is set to srTEST_FAIL and the method returns the value to the right of
    // the comma. The actual value is immaterial since by setting the status explicitly
    // we override the normal meaning of the return value.
    srASSERT_NE(h, 0), false;

    int rc = h->DoSomething();
    srEXPECT_EQ(rc, 0);

    return true;
}