Indicating Pass/Fail in a Test Method

From STRIDE Wiki
Jump to navigation Jump to search

Test Method Return Values

As described elsewhere, Stride test methods must conform to certain criteria. Among other things, these criteria specify allowable signatures for test methods.

Three distinct return return types are permitted, each with their own behavior with respect to how return values are interpreted by the Stride runtime in setting PASS/FAIL status.

Following is a table summarizing the details of this behavior and code examples illustrating the use of each return type.


Return Type Description How Return Value is Interpreted Default Status
void Most common return type Since there is no return, PASS/FAIL status is set in the body of the method using a

or a runtime call

  • If the method runs to completion without another status being set, the status is srTEST_PASS
  • If the method does not run to completion (due to crash or hang in software under test), the status is srTEST_INPROGRESS
integer type Integer types include:
  • int
  • short
  • char
  • long

may include signed/unsigned/const qualifiers

  • If you explicitly set the status in the method (using a test macro or a runtime call), this status will override the status that would otherwise be set by the return value
  • If you don't explicitly set the status in the method, the return value determines the status as follows:
    • Return is zero -> srTEST_PASS
    • Return is non-zero -> srTEST_FAIL
  • If the method runs to completion, the status is determined by the rules to the left (there is no default)
  • If the method does not run to completion (due to crash or hang in software under test), the status is srTEST_INPROGRESS
bool c++ only
  • If you explicitly set the status in the method (using a test macro or a runtime call), this status will override the status that would otherwise be set by the return value
  • If you don't explicitly set the status in the method, the return value determines the status as follows:
    • Return is true -> srTEST_PASS
    • Return is false -> srTEST_FAIL
  • If the method runs to completion, the status is determined by the rules to the left (there is no default)
  • If the method does not run to completion (due to crash or hang in software under test), the status is srTEST_INPROGRESS

Examples

The following example implementations validate the operation of a function named DoAdd() (not shown). Required header file declarations are assumed to be in place.

c++ examples are shown, but techniques are applicable to c as well.

void return

This is the return type that is most often used as it provides the most convenience.


#include <srtest.h>

/////////////////////////////
// shows use of a test macro to set status
/////////////////////////////
void MyTest::Test1()
{
    int sum = DoAdd(2, 2);
    // test macro validates condition and sets test status.
    srEXPECT_EQ(sum, 4);
}

/////////////////////////////
// shows use a a runtime call to set status
/////////////////////////////
void MyTest::Test2()
{
    int sum = DoAdd(3, 3);
    // set the status using a runtime call; much less convenient than
    // using the macro, but same result
    srTestStatus_e eStatus = (sum == 6) ? srTEST_PASS : srTEST_FAIL;
    testCase.SetStatus(eStatus, 0);
}

/////////////////////////////
// shows use of ASSERT macro to "short circuit" the test
/////////////////////////////
void MyTest::Test3()
{
    HANDLE h = GetResource();
    // if the resource is not acquired, there is no point in continuing; the
    // ASSERT macro validates and will set status to srTEST_FAIL and 
    // immediately return if the condition is not met
    srASSERT_NE(h, 0);

    int rc = h->DoSomething();
    srEXPECT_EQ(rc, 0);
}

/////////////////////////////
// shows default status
/////////////////////////////
void MyTest::Test4()
{
    // status is not set, so default srTEST_PASS is in force
}

integer return

#include <srtest.h>

/////////////////////////////
// shows use of return value to set status
/////////////////////////////
int MyTest::Test1()
{
    int sum = DoAdd(2, 2);
    // return 0 -> srTEST_PASS, non-zero -> srTEST_FAIL
    return (sum == 4) ? 0 : 1;
}

/////////////////////////////
// shows use of test macro to override return value
/////////////////////////////
int MyTest::Test2()
{
    int sum = DoAdd(3, 3);
    srEXPECT_EQ(sum, 6);
    // return value does not affect test status since the status
    // was set explicitly using a test macro above
    return 0;
}

/////////////////////////////
// shows use of ASSERT macro in test with non-void return type
/////////////////////////////
int MyTest::Test3()
{
    HANDLE h = GetResource();
    // if the resource is not acquired, there is no point in continuing; the
    // ASSERT macro validates and will set status to srTEST_FAIL and 
    // immediately return if the condition is not met
    //
    // Note the unusual syntax here; this allows the ASSERT macro (usually used where
    // void return type is in force) to be used here. If the ASSERT condition is not met,
    // the status is set to srTEST_FAIL and the method returns the value to the right of
    // the comma. The actual value is immaterial since by setting the status explicitly
    // we override the normal meaning of the return value.
    srASSERT_NE(h, 0), -1;

    int rc = h->DoSomething();
    srEXPECT_EQ(rc, 0);

    return 0;
}

bool return

#include <srtest.h>

bool MyTest::Test1()
{
    int sum = DoAdd(2, 2);
    // return true -> PASS, false -> FAIL
    return (sum == 4);
}

bool MyTest::Test2()
{
    int sum = DoAdd(3, 3);
    // 
    srEXPECT_EQ(sum, 6);
    // return value does not affect test status since the status
    // was set explicitly using a test macro above
    return true;
}

/////////////////////////////
// shows use of ASSERT macro in test with non-void return type
/////////////////////////////
bool MyTest::Test3()
{
    HANDLE h = GetResource();
    // if the resource is not acquired, there is no point in continuing; the
    // ASSERT macro validates and will set status to srTEST_FAIL and 
    // immediately return if the condition is not met
    //
    // Note the unusual syntax here; this allows the ASSERT macro (usually used where
    // void return type is in force) to be used here. If the ASSERT condition is not met,
    // the status is set to srTEST_FAIL and the method returns the value to the right of
    // the comma. The actual value is immaterial since by setting the status explicitly
    // we override the normal meaning of the return value.
    srASSERT_NE(h, 0), false;

    int rc = h->DoSomething();
    srEXPECT_EQ(rc, 0);

    return true;
}