Runtime Test Services: Difference between revisions

Revision as of 00:50, 27 November 2014

Runtime Test Services

The Runtime Test Services (declared in srtest.h) are a set of APIs in the STRIDE Runtime that facilitate the writing of target based test code. These APIs make up an optional portion of the STRIDE Runtime and can be used to communicate additional information about tests to the host based reporting mechanism. These APIs also allow target test code to create additional test suites and test cases dynamically at runtime.

There are two ways of accessing these APIs: through C-based functions, or through a C++ base class from which you may derive your C++ test class. These are discussed below in C Test Functions, and C++ Test Classes sections below.

Refer to the SCL Reference Guide or the Runtime Developer's Guide for more detailed information about any of these APIs.

C Test Functions

The following C APIs are provided:

srTestGetParam: returns an input parameters value.

srTestSuiteAddCase: creates an additional test case at runtime.
srTestSuiteAddAnnotation: creates a suite annotation at runtime.
srTestSuiteSetData: associates a custom name|value.

srTestCaseSetStatus: explicitly sets the status for the specified test case.
srTestCaseAddAnnotation: creates a test case annotation at runtime.
srTestSuiteSetData: associates a custom name|value pair with the specified test case.

srTestAnnotationAddComment: adds a comment to the specified annotation.

srTestGetParam

The srTestGetParam() function is used to get the value of a named parameter as a string.

srWORD srTestGetParam(const srCHAR * szName, srCHAR * szValue, srDWORD dwSize, const srCHAR * szDefValue);

Parameters	Type	Description
szName	Input	Pointer to a null-terminated string that represents the name of the parameter. Cannot be null.
szValue	Output	Pointer to a block of memory to store the value of the parameter with a maximum size of dwSize chars
dwSize	Input	Size in chars of the buffer pointed to by szValue
szDefValue	Input (optional)	Pointer to a null-terminated string that represents a default value in case the parameter is not specified. By setting this value to srNULL (or omitting it), you can use srTestGetParam() to test for the existence of a named parameter. If the named parameter doesn't exist, the function will return srERR_HANDLE_INVALID.

Return Value	Description
srOK	Success
srERR	Null or empty szName string passed in.
srERR_HANDLE_INVALID	The named parameter is not found and szDefValue is set to srNULL.

#include <srtest.h>

#define MAX_VALUE_LEN 128
void tfsuite_getParam(void)
{
  srCHAR szValue[MAX_VALUE_LEN] = {0};

  srWORD wRet = srTestGetParam("ParamName", szValue, MAX_VALUE_LEN, "default value");
}

#ifdef _SCL
#pragma scl_test_flist("testfunc", tfsuite_getParam)
#endif

srTestGetParamLong

The srTestGetParamLong() function is used to get the value of a named parameter as a long.

srLONG srTestGetParamLong(const srCHAR * szName, srLONG lDefValue);

Parameters	Type	Description
szName	Input	Pointer to a null-terminated string that represents the name of the parameter.
lDefValue	Input	Default value that is returned if the parameter is not found.

Return Value	Description
srLONG	Parameter value, or value specified by lDefValue if parameter isn't found.

#include <srtest.h>

void tfsuite_getParam(void)
{
  srLONG lVal = srTestGetParamLong("ParamName", -1);
}

#ifdef _SCL
#pragma scl_test_flist("testfunc", tfsuite_getParam)
#endif

srTestGetParamDouble

The srTestGetParamDouble() function is used to get the value of a named parameter as a double.

double srTestGetParamDouble(const srCHAR * szName, double dbDefValue);

Parameters	Type	Description
szName	Input	Pointer to a null-terminated string that represents the name of the parameter.
dbDefValue	Input	Default value that is returned if the parameter is not found.

Return Value	Description
double	Parameter value, or value specified by dbDefValue if parameter isn't found.

#include <srtest.h>

void tfsuite_getParam(void)
{
  double dbVal = srTestGetParamDouble("ParamName", 0.5772156649);
}

#ifdef _SCL
#pragma scl_test_flist("testfunc", tfsuite_getParam)
#endif

srTestSuiteAddCase

The srTestSuiteAddCase() routine is used to add a new test case to the specified test suite.

srTestCaseHandle_t srTestSuiteAddCase(const srCHAR * szName, const srCHAR * szDescr)

Parameters	Type	Description
szName	Input	Pointer to a null-terminated string that represents the name of the new test case. If null, the default host naming scheme will be used.
szDescr	Input	Pointer to a null-terminated string representing the description of the new test case. If null, description will be empty.

Return Value	Description
srTestCaseHandle_t	Handle of the newly created test case. srTEST_CASE_INVALID indicates failure to create a test case.

#include <srtest.h>
#include <stdio.h>

void tfsuite_addCase(void)
{
  for (int count = 0; count < 5; ++count)
  {
    char szName[25];
    sprintf(szName, "dynamic test case %d", count);
    srTestCaseHandle_t test = srTestSuiteAddCase(szName, "this is a dynamic test case");
    srTestCaseSetStatus(test, srTEST_PASS, 0); 
  }
}

#ifdef _SCL
#pragma scl_test_flist("testfunc", tfsuite_addCase)
#endif

srTestSuiteAddAnnotation

The srTestSuiteAddAnnotation() routine is used to add a new annotation to the specified test suite.

srTestAnnotationHandle_t srTestSuiteAddAnnotation(srTestAnnotationLevel_e eLevel,
                                                  const srCHAR * szName, 
                                                  const srCHAR * szFmtDescr, ...)

Parameters	Type	Description
eLevel	Input	Annotation level. srTEST_ANNOTATION_LEVEL_TRACE, srTEST_ANNOTATION_LEVEL_DEBUG, srTEST_ANNOTATION_LEVEL_INFO, srTEST_ANNOTATION_LEVEL_WARN, srTEST_ANNOTATION_LEVEL_ERROR, srTEST_ANNOTATION_LEVEL_FATAL
szName	Input	Pointer to a null-terminated string representing the name of the new annotation. If null, the default host naming scheme will be used.
szFmtDescr	Input	Pointer to a null-terminated string representing the description of the new annotation. Cannot be null.
...	Input (Optional)	Variable argument list to format szFmtDescr.

Return Value	Description
srTestAnnotationHandle_t	Handle of the newly created test annotation. srTEST_ANNOTATION_INVALID indicates failure to create a test annotation.

#include <srtest.h>

void tfsuite_addAnnotation(void) 
{
  for (int count = 0; count < 5; ++count)
  {
    char szName[25];
    sprintf(szName, "annotation %d", count);
    srTestAnnotationHandle_t annot = 
                     srTestSuiteAddAnnotation(srTEST_ANNOTATION_LEVEL_ERROR,
                                              szName,
                                              "description of annotation");
  }
}

#ifdef _SCL
#pragma scl_test_flist("testfunc", tfsuite_addAnnotation)
#endif

srTestSuiteSetData

The srTestSuiteSetData() routine is used to associate a custom name|value pair with a test suite.

srWORD srTestSuiteSetData(const srCHAR * szName, const srCHAR * szFmtValue, ...)

Parameters	Type	Description
szName	Input	Pointer to a null-terminated string representing the name of the custom pair. Cannot be null.
szFmtValue	Input	Pointer to a null-terminated string representing the value of the custom pair. Cannot be null.
...	Input (Optional)	Variable argument list to format szFmtValue.

Return Value	Description
srWORD	srOK on success, srERR_HANDLE_INVALID on invalid handle.

#include <srtest.h>

void tfcase_setData(void)
{
  srTestSuiteSetData("MyName", "my value");
}

#ifdef _SCL
#pragma scl_test_flist("testfunc", tfcase_setData)
#endif

srTestCaseSetStatus

The srTestCaseSetStatus() routine is used to set the result of test case.

srWORD srTestCaseSetStatus(srTestCaseHandle_t tTestCase, srTestStatus_e eStatus, srDWORD dwDuration)

Parameters	Type	Description
tTestCase	Input	Handle to a test case. srTEST_CASE_DEFAULT can be used for the default test case.
eStatus	Input	Result of the test. Possible values are: srTEST_FAIL srTEST_PASS srTEST_NOTINUSE srTEST_INPROGRESS srTEST_DONE - applicable to dynamic cases - sets the status to pass unless already set to fail or not-in-use
dwDuration	Input	The duration of the test in clock ticks. Using "0" allows the STRIDE Runtime (Intercept Module) to set the time automatically.

Return Value	Description
srWORD	srOK on success, srERR_HANDLE_INVALID on invalid handle.

#include <srtest.h>

void tfcase_setStatus(void)
{
  srTestCaseSetStatus(srTEST_CASE_DEFAULT, srTEST_PASS, 0);
}

#ifdef _SCL
#pragma scl_test_flist("testfunc", tfcase_setStatus)
#endif

srTestCaseSetStatusEx

The srTestCaseSetStatusEx() routine is used to set the result of test case and allow specification of an extendedFailureCode.

srWORD srTestCaseSetStatus(srTestCaseHandle_t tTestCase, srTestStatus_e eStatus, srDWORD dwDuration, srLONG lExtendedFailureCode)

Parameters	Type	Description
tTestCase	Input	Handle to a test case. srTEST_CASE_DEFAULT can be used for the default test case.
eStatus	Input	Result of the test.
dwDuration	Input	The duration of the test in clock ticks. Using "0" allows the STRIDE Runtime (Intercept Module) to set the time automatically.
lExtendedFailureCode	Input	The Stride framework uses the extendedFailureCode to capture the numeric results of test method when the test method fails via a numeric (non-void, nonbool) return type.

Return Value	Description
srWORD	srOK on success, srERR_HANDLE_INVALID on invalid handle.

#include <srtest.h>

void tfcase_setStatusEx(void)
{
  srTestCaseSetStatusEx(srTEST_CASE_DEFAULT, srTEST_FAIL, 0, -5);
}

#ifdef _SCL
#pragma scl_test_flist("testfunc", tfcase_setStatusEx)
#endif

srTestCaseAddAnnotation

The srTestCaseAddAnnotation() routine is used to add a new annotation to the specified test case.

srTestAnnotationHandle_t srTestCaseAddAnnotation(rTestCaseHandle_t tTestCase, srTestAnnotationLevel_e eLevel, const srCHAR * szName, const srCHAR * szFmtDescr, ...)

Parameters	Type	Description
tTestCase	Input	Handle to the parent test case to which new test annotation is to be added. srTEST_CASE_DEFAULT can be used for the default test case.
eLevel	Input	Annotation level. srTEST_ANNOTATION_LEVEL_TRACE, srTEST_ANNOTATION_LEVEL_DEBUG, srTEST_ANNOTATION_LEVEL_INFO, srTEST_ANNOTATION_LEVEL_WARN, srTEST_ANNOTATION_LEVEL_ERROR, srTEST_ANNOTATION_LEVEL_FATAL
szName	Input	Pointer to a null-terminated string representing the name of the new annotation. If null, the default host naming scheme will be used.
szFmtDescr	Input	Pointer to a null-terminated string representing the description of the new annotation. Cannot be null.
...	Input (Optional)	Variable argument list to format szFmtDescr.

Return Value	Description
srTestAnnotationHandle_t	Handle of the newly created test annotation. srTEST_ANNOTATION_INVALID indicates failure to create a test annotation.

#include <srtest.h>

void tfcase_addAnnotation(void) 
{
  for (int count = 0; count < 5; ++count)
  {
    char szName[25];
    sprintf(szName, "annotation %d", count);
    srTestAnnotationHandle_t annot = 
                     srTestCaseAddAnnotation(srTEST_CASE_DEFAULT,
                                             srTEST_ANNOTATION_LEVEL_ERROR,
                                             szName,
                                             "description of annotation");
  }
}

#ifdef _SCL
#pragma scl_test_flist("testfunc", tfcase_addAnnotation)
#endif

srTestCaseSetData

The srTestCaseSetData() routine is used to associate a custom name|value pair with a test case.

srWORD srTestCaseSetData(srTestCaseHandle_t tTestCase, const srCHAR * szName, const srCHAR * szFmtValue, ...);

Parameters	Type	Description
tTestCase	Input	Handle to a test case. srTEST_CASE_DEFAULT can be used for the default test case.
szName	Input	Pointer to a null-terminated string representing the name of the custom pair. Cannot be null.
szFmtValue	Input	Pointer to a null-terminated string representing the value of the custom pair. Cannot be null.
...	Input (Optional)	Variable argument list to format szFmtValue.

Return Value	Description
srWORD	srOK on success, srERR_HANDLE_INVALID on invalid handle.

#include <srtest.h>

void tfcase_setData(void)
{
  srTestCaseSetData(srTEST_CASE_DEFAULT, "MyName", "my value");
}

#ifdef _SCL
#pragma scl_test_flist("testfunc", tfcase_setData)
#endif

srTestAnnotationAddComment

The srTestAnnotationAddComment() routine is used to add a comment (aka a log) to a test annotation.

srWORD srTestAnnotationAddComment(srTestAnnotationHandle_t tTestAnnotation, const srCHAR * szLabel, const srCHAR * szFmtComment, ...)

Parameters	Type	Description
tTestAnnotation	Input	Handle to a test annotation.
szLabel	Input	Pointer to a null-terminated string for the label. If null, the default label will be applied.
szFmtComment	Input	Pointer to a null-terminated string representing the new comment. Cannot be null.
...	Input (Optional)	Variable argument list to format szFmtComment.

Return Value	Description
srWORD	srOK on success, srERR on null string or invalid formatted string, srERR_HANDLE_INVALID on invalid handle, srERR_STR_TRUNCATED on truncated string.

#include <srtest.h>

void tfsuiteAnnotation_addComment(void) 
{
  srTestAnnotationHandle_t annot = 
                           srTestSuiteAddAnnotation(srTEST_ANNOTATION_LEVEL_ERROR,
                                                    "annot",
                                                    "annot description");
  srTestAnnotationAddComment(annot,
                             srNULL,
                             "this comment should print %s", "A STRING");
}

#ifdef _SCL
#pragma scl_test_flist("testfunc", tfsuiteAnnotation_addComment)
#endif

C++ Test Classes

The Runtime Test Services C APIs work equally well from C test functions and C++ test classes. If, however, you are using C++ it might be more convinient for you to derive your C++ test classes from the Runtime Test Services C++ base class, srTest. That way you will have access to a set of simpler to use C++ classes.

The following C++ classes are provided:

class srTest - base test class
class srTestCase - represents a test case
class srTestAnnotation - represents a test annotation

class srTest

The srTest class provides one member object testCase of class srTestCase and the folowing methods:

method GetParam

This overload of the GetParam() method is used to get the value of a named parameter as a string.

static srWORD GetParam(const srCHAR * name, srCHAR * value, srDWORD size, const srCHAR * defval = srNULL)

Parameters	Type	Description
name	Input	Pointer to a null-terminated string that represents the name of the parameter. Cannot be null.
value	Output	Pointer to a block of memory to store the value of the parameter with a maximum size of size chars
size	Input	Size in chars of the buffer pointed to by value
defvalue	Input (optional)	Pointer to a null-terminated string that represents a default value in case the parameter is not specified. By setting this value to srNULL (or omitting it), you can use GetParam() to test for the existence of a named parameter. If the named parameter doesn't exist, the function will return srERR_HANDLE_INVALID.

Return Value	Description
srWORD	srOK on success, error otherwise.

#include <srtest.h>

class srtest_class : public stride::srTest
{
public:
  void getParam()
  {
    GetParam("ParamName", szValue, MAX_VALUE_LEN, "default value");
  }
};

#ifdef _SCL
#pragma scl_test_class(srtest_class)
#endif

method GetParam

This overload of the GetParam() method is used to get the value of a named parameter as a long.

static long GetParam(const srCHAR * name, srLONG defval = 0)

Parameters	Type	Description
name	Input	Pointer to a null-terminated string that represents the name of the parameter.
defvalue	Input	Default value that is returned if the parameter is not found.

Return Value	Description
srLONG	Parameter value, or value specified by defvalue if parameter isn't found.

#include <srtest.h>

class srtest_class : public stride::srTest
{
public:
  void getParam()
  {
    srLONG lValue = GetParam("ParamName", -1);
  }
};

#ifdef _SCL
#pragma scl_test_class(srtest_class)
#endif

method GetParam

This overload of the GetParam() method is used to get the value of a named parameter as a double.

static double GetParam(const srCHAR * name, const double& defval)

Parameters	Type	Description
name	Input	Pointer to a null-terminated string that represents the name of the parameter.
defvalue	Input	Default value that is returned if the parameter is not found.

Return Value	Description
double	Parameter value, or value specified by defvalue if parameter isn't found.

#include <srtest.h>

class srtest_class : public stride::srTest
{
public:
  void getParam()
  {
    double dbVal = GetParam("ParamName",  0.5772156649);
  }
};

#ifdef _SCL
#pragma scl_test_class(srtest_class)
#endif

method AddCase

The AddCase method is used to new test case to the test suite.

srTestCase AddCase(const srCHAR * name, const srCHAR * descr = srNULL)

Parameters	Type	Description
name	Input	Pointer to null-terminated string that represents the name of the new test case. If empty, the default host naming scheme will be used.
descr	Input (Optional)	Pointer to null-terminated string representing the description of the new test case. If empty, description will be blank.

Return Value	Description
srTestCase	Newly created test case instance.

Example

#include <srtest.h>
#include <sstream>

class srtest_class : public stride::srTest
{
public:
  void suiteAddSuite()
  {
    const std::string prefix("dynamic test case ");
    for (int count = 0; count < 5; ++count)
    {
      std::stringstream strm;
      strm << prefix << count;
      stride::srTestCase tc = AddCase(strm.str().c_str());
      tc.SetStatus(srTEST_PASS);
    }
  }
};

#ifdef _SCL
#pragma scl_test_class(srtest_class)
#endif

method AddAnnotation

The AddAnnotation method is used to add a new test annotation to the test suite.

srTestAnnotation AddAnnotation(srTestAnnotationLevel_e level, const srCHAR * name, const srCHAR * descr, ...)

Parameters	Type	Description
level	Input	Annotation level. srTEST_ANNOTATION_LEVEL_TRACE, srTEST_ANNOTATION_LEVEL_DEBUG, srTEST_ANNOTATION_LEVEL_INFO, srTEST_ANNOTATION_LEVEL_WARN, srTEST_ANNOTATION_LEVEL_ERROR, srTEST_ANNOTATION_LEVEL_FATAL
name	Input	Pointer to null-terminated string that represents the name of the new annotation. If empty, the default host naming scheme will be used.
descr	Input	Pointer to null-terminated string representing the description of the new annotation. Cannot be null.

Return Value	Description
srTestAnnotation	Newly created annotation instance.

#include <srtest.h>
#include <sstream>

class srtest_class : public stride::srTest
{
public:
  void suiteAddAnnotation()
  {
    for (int count = 0; count < 5; ++count)
    {
      std::stringstream strmName;
      std::stringstream strmDescr;
      strmName << "annotation " << count;
      strmDescr << "description of annotation " << count;
      stride::srTestAnnotation ta = AddAnnotation(srTEST_ANNOTATION_LEVEL_INFO,
                                                  strmName.str().c_str(),
                                                  strmDescr.str().c_str());
    }
  }
};

#ifdef _SCL
#pragma scl_test_class(srtest_class)
#endif

method SetData

The SetData method is used to associate a custom name|value pair with the test suite.

srWORD SetData(const srCHAR * name, const srCHAR * value, ...)

Parameters	Type	Description
name	Input	Pointer to a null-terminated string representing the name of the custom pair. Cannot be null.
value	Input	Pointer to a null-terminated string representing the value of the custom pair. Cannot be null.

Return Value	Description
srWORD	srOK on success, srERR_HANDLE_INVALID on invalid handle.

Example

#include <srtest.h>
#include <sstream>

class srtest_class : public stride::srTest
{
public:
  void suiteSetData()
  {
      SetData("MyName", "my value");
  }
};

#ifdef _SCL
#pragma scl_test_class(srtest_class)
#endif

class srTestCase

The srTestCase class provides the following methods:

method SetStatus

The SetStatus method is used to set the result of the default test case.

srWORD SetStatus(srTestStatus_e status, srDWORD duration = 0)

Parameters	Type	Description
status	Input	Result of the test. Possible values are: srTEST_FAIL srTEST_PASS srTEST_NOTINUSE srTEST_INPROGRESS srTEST_DONE - applicable to dynamic cases - sets the status to pass unless already set to fail or not-in-use
duration	Input (Optional)	The duration of the test in clock ticks. The default is 0.

Return Value	Description
srWORD	srOK on success, srERR_HANDLE_INVALID on invalid handle.

#include <srtest.h>

class srtest_class : public stride::srTest
{
public:
  void caseSetStatus()
  {
    testCase.SetStatus(srTEST_NOTINUSE);
  }
};

#ifdef _SCL
#pragma scl_test_class(srtest_class)
#endif

method AddAnnotation

The AddAnnotation method is used to add a new test annotation to the test case.

srTestAnnotation AddAnnotation(srTestAnnotationLevel_e level, const srCHAR * name, const srCHAR * descr, ...)

Parameters	Type	Description
level	Input	Annotation level. srTEST_ANNOTATION_LEVEL_TRACE, srTEST_ANNOTATION_LEVEL_DEBUG, srTEST_ANNOTATION_LEVEL_INFO, srTEST_ANNOTATION_LEVEL_WARN, srTEST_ANNOTATION_LEVEL_ERROR, srTEST_ANNOTATION_LEVEL_FATAL
name	Input	Pointer to null-terminated string that represents the name of the new annotation. If null, the default host naming scheme will be used.
descr	Input	Pointer to null-terminated string representing the description of the new annotation. Cannot be null.

Return Value	Description
srTestAnnotation	Newly created annotation instance.

#include <srtest.h>
#include <sstream>

class srtest_class : public stride::srTest
{
public:
  void caseAddAnnotation()
  {
    for (int count = 0; count < 5; ++count)
    {
      std::stringstream strmName;
      std::stringstream strmDescr;
      strmName << "annotation " << count;
      strmDescr << "description of annotation " << count;
      stride::srTestAnnotation ta = 
                 testCase.AddAnnotation(srTEST_ANNOTATION_LEVEL_INFO,
                                         strmName.str().c_str(),
                                         strmDescr.str().c_str());
    }
  }
};

#ifdef _SCL
#pragma scl_test_class(srtest_class)
#endif

method SetData

The SetData method is used to associate a custom name|value pair with the test case.

srWORD SetData(const srCHAR * name, const srCHAR * value, ...)

Parameters	Type	Description
name	Input	Pointer to a null-terminated string representing the name of the custom pair. Cannot be null.
value	Input	Pointer to a null-terminated string representing the value of the custom pair. Cannot be null.

Return Value	Description
srWORD	srOK on success, srERR_HANDLE_INVALID on invalid handle.

#include <srtest.h>

class srtest_class : public stride::srTest
{
public:
  void caseSetData()
  {
    testCase.SetData("MyName", "my value");
  }
};

#ifdef _SCL
#pragma scl_test_class(srtest_class)
#endif

class srTestAnnotation

The srTestAnnotation class provides the following methods:

method AddComment

The AddComment method is used to add a comment to the test annotation.

srWORD AddComment(const srCHAR * label, const srCHAR * comment, ...)

Parameters	Type	Description
label	Input	Pointer to null-terminated string for the label. If null, the default label will be applied.
comment	Input	Pointer to null-terminated string representing the new comment. Cannot be empty.

Return Value	Description
srWORD	srOK on success, srERR on null string or invalid formatted string, srERR_HANDLE_INVALID on invalid handle, srERR_STR_TRUNCATED on truncated string.

#include <srtest.h>

class srtest_class : public stride::srTest
{
public:
  void suiteAnnotationAddComment()
  {
    stride::srTestAnnotation ta = 
                 testSuite.AddAnnotation(srTEST_ANNOTATION_LEVEL_INFO,
                                         "annot",
                                         "annot description");
    ta.AddComment("this comment on annotation should print %s", "A STRING");
  }
};

#ifdef _SCL
#pragma scl_test_class(srtest_class)
#endif

Runtime Test Services: Difference between revisions

Revision as of 00:50, 27 November 2014

Contents