Test Units Overview: Difference between revisions

From STRIDE Wiki
Jump to navigation Jump to search
 
(206 intermediate revisions by 6 users not shown)
Line 1: Line 1:
== Introduction  ==
== What are STRIDE Test Units? ==


STRIDE enables testing of C/C++ code through the use of xUnit-style test units. Test units can be written by developers, captured using an SCL pragma, and executed from the host. STRIDE facilitates the execution of some or all of the test units by automatically creating entry points for the execution of test units on the target.  
'''STRIDE Test Units''' is a general term for [http://en.wikipedia.org/wiki/XUnit xUnit-style] test modules running within the STRIDE runtime framework. These tests--written in C and C++--are compiled and linked with your embedded software and run in-place on your target hardware. They are suitable for both developer unit testing as well as end-to-end integration testing.


== Using test units  ==
An external [[Stride Runner|Test Runner]] is provided which controls the execution of the tests and publishes test results to the local file system and optionally to S2's Internet [[STRIDE Test Space]].


=== Prerequisites  ===
== Test Unit Features ==
In all cases, STRIDE Test Units provide the following capabilities typical of all [http://en.wikipedia.org/wiki/XUnit xUnit-style] testing frameworks:
* Specification of a test as a test method
* Aggregation of individual tests into test suites which form execution and reporting units
* Specification of expected results within test methods (typically by using one or more [[Test Code Macros]])
* Test fixturing (optional setup and teardown)
* Test parametrization (optional constructor/initialization parameters)
* Automated execution
* Automated results report generation


see [[Host_Installation#Third Party Components|Perl requirements]].  
=== Unique STRIDE Test Unit Features ===
In addition, STRIDE Test Units offer these unique features:
; On-Target Execution
: Tests execute on the target hardware in a true operational environment. Execution and reporting is controlled from a remote desktop (Windows, Linux or FreeBSD) host
; Dynamic Test and Suite Generation
: Test cases and suites can be created and manipulated at runtime
; [[Using Test Doubles|Test Doubles]]
: Dynamic runtime function substitution to implement on-the-fly mocks, stubs, and doubles
; [[Test Point Testing in C/C++|Behavior Testing]] (Test Points)
: Support for testing of asynchronous activities occurring in multiple threads
; Multiprocess Testing Framework
: Support for testing across multiple processes running simultaneously on the target
; Automatic Timing Data Collection
: Duration are automatically measured for each test case.
; Automatic Results Publishing to Local Disk and Internet
: Automatic publishing of test results to [[STRIDE Test Space]]


===How to get started (Overview)===
[[Category:Test Units]]
 
The required steps to get started with writing test units are as follows:
 
<ol>
<li>Create a new Studio workspace (or open an existing one).</li>
<li>Set the workspace to compile in C++ mode (In Studio, choose Tools->Settings->Compile as Cpp).</li>
<li>Write a test unit. You may create a C++ test class or a C test function as your test unit. Click '''[[Test_Units#Test_Unit_Requirements|here]]''' for more information on creating test units.</li>
To implement a test unit as a test class:
<source lang=cpp>
#include <srtest.h>
 
class Simple {
public:
    int test1(void) { return  0;} // PASS
    int test2(void) { return 23;} // FAIL <>0
};
 
#pragma scl_test_class(Simple)
</source>
Or, as a test function:
<source lang=cpp>
#ifdef __cplusplus
extern "C" {
#endif
 
int test1(void)
{
    return 0; // PASS
}
 
int test1(void)
{
    return 23; // FAIL <>0
}
 
#pragma scl_test_flist("Simple", test1, test2)
 
#ifdef __cplusplus
}
#endif
</source>
 
<li>Compile the workspace & review the '''Simple''' interface in the Studio Interface tab</li>
<li>Create a script to generate the Intercept Module(IM) '''after''' the compilation step.
:For the simple STUB generation required for test unit execution, you can use the following code</li>
<source lang=perl>
  use strict;
  use Win32::OLE qw(in);
  Win32::OLE->Option(Warn => 3);
  my $intercept = $main::studio->Workspace->Intercept;   
  $intercept->{Path} = $main::studio->Workspace->Path;
  $intercept->{Name} = $main::studio->Workspace->Name;
  map {$intercept->Item($_)->{Stub} = 1} (0..($intercept->Count - 1));
  $intercept->Create();
</source>
<li>Optionally add custom scripts to automate the building and executing your application. Refer to [[Using Frameworks]] for more information regarding building and executing tests with a target device.</li>
<li>Ensure that the Studio workspace include path contains the location to all of your test unit declaration (header) files.</li>
<li>Once you have created one or more test units, ensure the following:
:* Workspace is compiled and saved
:* Intercept Module is generated (Stubs for all Test Units)
:* Target application re-built
:* Target application downloaded & started
:* STRIDE Connected to Target
</li>
 
<li>If your application is running, you can start executing test units.
:* You can test-execute individual test units interactively using the '''Studio interface''' view. To do this, open the user interface view corresponding to the test unit you would like to execute, then call it. The return values will indicate how many tests produced each of four (4) result types. Furthermore, the input to the entry point will allow you to select all methods for execution (the default) or individual methods via a dropdown list of enumerated values.
:* Once you are confident that the test units are behaving as expected, you can generate one or more execution scripts using the '''Script Wizard'''.  Sample '''[[templates]]''' for executing test unit entry points are provided in the %STRIDE_DIR%\templates\Script Wizard directory.
:* When writing scripts to execute test units, the [[AutoScript#ascript.TestUnits|AutoScript TestUnits]] collection is a powerful tool for test unit execution and reporting, and for obtaining test results.</li>
<li>For integration with larger regression test workspaces, we recommend that developers check in their test unit code and, optionally, the template-generated scripts that can be used to execute their test units.</li>
</ol>
 
===Pragmas for Test Units===
STRIDE supports three pragmas for capturing and qualifying test units:
 
*'''<code>scl_test_class ( class )</code>''': Declares a test class as captured. Once captured, STRIDE will generate the appropriate code for executing the test methods in the class. See the [[scl_test_class|scl_test_class page]] for more information.
*'''<code>scl_test_flist ( test-unit-name , test-function-name { , test-function-name } )</code>''': Declares a test unit as captured, by the specified test-unit-name. One or more functions (test methods) are associated with the test unit. Once captured, STRIDE will generate the appropriate code for executing the associated test methods. See the [[scl_test_flist|scl_test_flist page]] for more information.
*'''<code>scl_test_setup ( test-unit-name , method-name )</code>''': [optional] Declares a member method to be a setup fixture for the test unit. If specified, the setup method will be called before the execution of each test method. See the [[scl_test_setup|scl_test_setup page]] for more information.
*'''<code>scl_test_teardown ( test-unit-name , method-name )</code>''': [optional] Declares a member method to be a teardown fixture for the test unit. If specified, the teardown method will be called after the execution of each test method. See the [[scl_test_teardown|scl_test_teardown page]] for more information.
 
== Test Unit Requirements  ==
 
Several variations on typical xUnit-style test units are supported. The additional supported features include:
 
*Test status can be set using STRIDE Runtime APIs ''or'' by specifying simple return types for test methods.
*Simple return types: 0 = PASS; &lt;&gt; 0 = FAIL
*void return type with no explict status setting is assumed PASS
*Test writers can create additional child suites and tests at runtime by using Runtime APIs.
*We do not rely on exceptions for reporting of status.
 
The STRIDE test class framework has the following requirements of each test class:
 
*The test class must have a suitable default (no-argument) constructor.
*The test class must have one or more public methods suitable as test methods. Allowable test methods always take no arguments (void) and return either void or simple integer types (int, short, long, char or bool). At this time, we do not allow typedef types or macros for the return values specification.
*the scl_test_class pragma must be applied to the class.
 
 
=== Simple example using return values for status  ===
==== Using a Test Class ====
 
  #include &lt;srtest.h&gt;
 
  class Simple {
  public:
    int tc_Int_ExpectPass(void) {return 0;}
    int tc_Int_ExpectFail(void) {return -1;}
    bool tc_Bool_ExpectPass(void) {return true;}
    bool tc_Bool_ExpectFail(void) {return false;}
  };
  #ifdef _SCL
  #pragma scl_test_class(Simple)
  #endif
 
==== Using a Test Function ====
  #include &lt;srtest.h&gt;
 
  #ifdef __cplusplus
  extern "C" {
  #endif
 
  int tf_Int_ExpectPass(void) {return 0;}
  int tf_Int_ExpectFail(void) {return -1;}
  bool tf_Bool_ExpectPass(void) {return true;}
  bool tf_Bool_ExpectFail(void) {return false;}
  #ifdef _SCL
  #pragma scl_test_flist("Simple", tf_Int_ExpectPass, tf_Int_ExpectFail, tf_Bool_ExpectPass, tf_Bool_ExpectFail)
  #endif
 
  #ifdef __cplusplus
  }
  #endif
 
=== Simple example using runtime test service APIs  ===
==== Using a Test Class ====
  #include &lt;srtest.h&gt;
 
  class RuntimeServices_basic {
  public:
    void tc_ExpectPass(void)
    {
        srTestCaseAddComment(srTEST_CASE_DEFAULT, "this test should pass");
        srTestCaseSetStatus(srTEST_CASE_DEFAULT, srTEST_PASS, 0);
    }
    void tc_ExpectFail(void)
    {
        srTestCaseAddComment(srTEST_CASE_DEFAULT, "this test should fail");
        srTestCaseSetStatus(srTEST_CASE_DEFAULT, srTEST_FAIL, 0);
    }
    void tc_ExpectInProgress(void)
    {
        srTestCaseAddComment(srTEST_CASE_DEFAULT, "this test should be in progress");
    }
  };
  #ifdef _SCL
  #pragma scl_test_class(RuntimeServices_basic)
  #endif
 
==== Using a Test Function ====
  #include &lt;srtest.h&gt;
 
  #ifdef __cplusplus
  extern "C" {
  #endif
 
  void tf_ExpectPass(void)
  {
      srTestCaseAddComment(srTEST_CASE_DEFAULT, "this test should pass");
      srTestCaseSetStatus(srTEST_CASE_DEFAULT, srTEST_PASS, 0);
  }
  void tf_ExpectFail(void)
  {
      srTestCaseAddComment(srTEST_CASE_DEFAULT, "this test should fail");
      srTestCaseSetStatus(srTEST_CASE_DEFAULT, srTEST_FAIL, 0);
  }
  void tf_ExpectInProgress(void)
  {
      srTestCaseAddComment(srTEST_CASE_DEFAULT, "this test should be in progress");
  }
  #ifdef _SCL
  #pragma scl_test_flist("RuntimeServices_basic", tf_ExpectPass, tf_ExpectFail, tf_ExpectInProgress)
  #endif
  #ifdef __cplusplus
  }
  #endif
 
=== Simple example using srTest base class  ===
 
  #include &lt;srtest.h&gt;
 
  class MyTest&nbsp;: public stride::srTest {
  public:
    void tc_ExpectPass(void)
    {
        testCase.AddComment("this test should pass");
        testCase.SetStatus(srTEST_PASS, 0);
    }
    void tc_ExpectFail(void)
    {
        testCase.AddComment("this test should fail");
        testCase.SetStatus(srTEST_FAIL, 0);
    }
    void tc_ExpectInProgress(void)
    {
        testCase.AddComment("this test should be in progress");
    }
    int tc_ChangeMyName(void)
    {
        testCase.AddComment("this test should have name = MyChangedName");
        testCase.SetName("MyChangedName");
        return 0;
    }
    int tc_ChangeMyDescription(void)
    {
        testCase.AddComment("this test should have a description set");
        testCase.SetDescription("this is my new description");
        return 0;
    }
  };
  #ifdef _SCL
  #pragma scl_test_class(MyTest)
  #endif
 
== Runtime Test Services  ==
 
The Runtime Test Services (declared in srTest.h) are a set of APIs in the STRIDE Runtime that facilitate the writing of target based test code. These APIs make up an optional portion of the STRIDE Runtime and can be used to communicate additional information about tests to the host based reporting mechanism. These APIs also allow target test code to create additional test suites and test cases dynamically at runtime.
 
There are 2 alternate ways of accessing these APIs: through C-based functions, or through a C++ base class from which you may derive your C++ test class. These are discussed below in C Test Functions, and C++ Test Classes sections below.
 
=== C Test Functions ===
 
The following C APIs are provided:
 
*'''<code>srTestSuiteAddSuite</code>''': creates an additional sub-suite at runtime.
*'''<code>srTestSuiteSetName</code>''': sets the name of the specified suite.
*'''<code>srTestSuiteSetDescription</code>''': sets the description of the specified suite.
*'''<code>srTestSuiteAddTest</code>''': creates an additional test case at runtime.
*'''<code>srTestCaseSetName</code>''': sets the name of the specified test case.
*'''<code>srTestCaseSetDescription</code>''': sets the description of the specified test case.
*'''<code>srTestCaseAddComment</code>''': adds a comment to the specified test case.
*'''<code>srTestCaseSetStatus</code>''': explicitly sets the status for the specified test case.
*'''<code>srTestSuiteAddAnnotation</code>''': creates an annotation at runtime.
*'''<code>srTestAnnotationAddComment</code>''': adds a comment to the specified annotation.
 
 
==== srTestSuiteAddSuite ====
'''Prototype'''
  srTestSuiteHandle_t srTestSuiteAddSuite(srTestSuiteHandle_t tParent, const srCHAR * szName)
 
'''Description'''
The srTestSuiteAddSuite() routine is used to add a new test suite to the specified test suite.
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;" 
| '''Parameters'''
| '''Type'''
| '''Description'''
|-
| tParent
| Input
| Handle to the parent test suite to which new test suite is to be added. srTEST_SUITE_DEFAULT can be used for the default test suite.
|-
| szName
| Input
| Pointer to a null-terminated string that represents the name of test suite. If null, the default host naming scheme will be used.
|}
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;"
| '''Return Value'''
| ''' Description'''
|-
|srTestSuiteHandle_t
| Handle of the newly created test suite. srTEST_SUITE_INVALID indicates failure to create test suite.
|}
<br>
'''Example'''
  #include "srtest.h"
  void tfsuite_addSuite(void)
  {
    srTestSuiteHandle_t subSuite =
    srTestSuiteAddSuite(srTEST_SUITE_DEFAULT, "tf Sub Suite");
  }<br>
  #ifdef _SCL
  #pragma scl_test_flist(“testfunc”, tfsuite_addSuite)
  #endif
 
 
==== srTestSuiteSetName ====
'''Prototype'''
  srWORD srTestSuiteSetName(srTestSuiteHandle_t tTestSuite, const srCHAR * szName)
 
'''Description'''
The srTestSuiteSetName() routine is used to set the name of the specified test suite.
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;" 
| '''Parameters'''
| '''Type'''
| '''Description'''
|-
| tTestSuite
| Input
| Handle to a test suite. srTEST_SUITE_DEFAULT can be used for the default test suite.
|-
| szName
| Input
| Pointer to a null-terminated string representing the name of test suite. Cannot be null.
|}
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;"
| '''Return Value'''
| ''' Description'''
|-
| srOK
| Success
|-
| srERR
| Null string passed in.
|-
| srTEST_ERR_HANDLE_INVALID
| Invalid handle passed in.
|-
| srTEST_WARN_STR_TRUNCATED
| String passed in was too large and was truncated.
|}
<br>
'''Example'''
  #include "srtest.h"
  void tfsuite_setName(void)
  {
    srTestSuiteSetName(srTEST_SUITE_DEFAULT, "Setting name for default suite");
  } <br>
  #ifdef _SCL
  #pragma scl_test_flist(“testfunc”, tfsuite_setName)
  #endif
 
==== srTestSuiteSetDescription ====
'''Prototype'''
  srWORD srTestSuiteSetDescription(srTestSuiteHandle_t tTestSuite, const srCHAR * szDescr)
 
'''Description'''
The srTestSuiteSetDescription() routine is used to set the description of the specified test
suite.
 
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;" 
| '''Parameters'''
| '''Type'''
| '''Description'''
|-
| tTestSuite
| Input
| Handle to a test suite. srTEST_SUITE_DEFAULT can be used for the default test suite.
|-
| szDescr
| Input
| Pointer to a null-terminated string representing the description of test suite. Cannot be null.
|}
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;"
| '''Return Value'''
| ''' Description'''
|-
| srOK
| Success
|-
| srERR
| Null string passed in.
|-
| srTEST_ERR_HANDLE_INVALID
| Invalid handle passed in.
|-
| srTEST_WARN_STR_TRUNCATED
| String passed in was too large and was truncated.
|}
<br>
'''Example'''
  #include "srtest.h"
  void tfsuite_setDescription(void)
  {
    srTestSuiteSetDescription(srTEST_SUITE_DEFAULT,
                              "Setting description for default suite");
  } <br>
  #ifdef _SCL
  #pragma scl_test_flist(“testfunc”, tfsuite_setDescription)
  #endif
 
==== srTestSuiteAddTest ====
'''Prototype'''
  srTestCaseHandle_t srTestSuiteAddTest(srTestSuiteHandle_t tParent, const srCHAR * szName)
 
'''Description'''
The srTestSuiteAddTest() routine is used to add a new test case to the specified test suite.
 
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;" 
| '''Parameters'''
| '''Type'''
| '''Description'''
|-
| tParent
| Input
| Handle to the parent test suite to which new test case is to be added. srTEST_SUITE_DEFAULT can be used for the default test suite.
|-
| szName
| Input
| Pointer to a null-terminated string that represents the name of test case. If null, the default host naming scheme will be used.
|}
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;"
| '''Return Value'''
| ''' Description'''
|-
| srTestCaseHandle_t
| Handle of the newly created test case. srTEST_CASE_INVALID indicates failure to create test case.
|}
<br>
'''Example'''
  #include "srtest.h"
  #include <sstream>
  void tfsuite_addTest(void)
  {
    const std::string prefix("dynamic test case ");
    for (int count = 0; count < 5; ++count)
    {
      std::stringstream strm;
      strm << prefix << count;
      srTestCaseHandle_t test = srTestSuiteAddTest(srTEST_SUITE_DEFAULT,
                                                  strm.str().c_str());
      srTEST_ADD_COMMENT_WITH_LOCATION(test, "this is a dynamic test case");
    }
  }<br>
  #ifdef _SCL
  #pragma scl_test_flist(“testfunc”, tfsuite_addTest)
  #endif
 
==== srTestCaseSetName ====
'''Prototype'''
  srWORD srTestCaseSetName(srTestCaseHandle_t tTestCase, const srCHAR *szName)
 
'''Description'''
The srTestCaseSetName() routine is used to set set the name of the specified test case.
 
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;" 
| '''Parameters'''
| '''Type'''
| '''Description'''
|-
| tTestCase
| Input
| Handle to a test case. srTEST_CASE_DEFAULT can be used for the default test case.
|-
| szName
| Input
| Pointer to a null-terminated string representing the name of test case. Cannot be null.
|}
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;"
| '''Return Value'''
| ''' Description'''
|-
| srOK
| Success
|-
| srERR
| Null string passed in.
|-
| srTEST_ERR_HANDLE_INVALID
| Invalid handle passed in.
|-
| srTEST_WARN_STR_TRUNCATED
| String passed in was too large and was truncated.
|}
<br>
'''Example'''
  #include "srtest.h"
  #include <sstream>
  void tfcase_setName(void)
  {
    srTestCaseSetName(srTEST_CASE_DEFAULT, "Setting name for default case");
  } <br>
  #ifdef _SCL
  #pragma scl_test_flist(“testfunc”, tfcase_setName)
  #endif
 
==== srTestCaseSetDescription ====
'''Prototype'''
  srWORD srTestCaseSetDescription(srTestCaseHandle_t tTestCase, const srCHAR * szDescr)
 
'''Description'''
The srTestCaseSetDescription() routine is used to set the description of the specified test
case.
 
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;" 
| '''Parameters'''
| '''Type'''
| '''Description'''
|-
| tTestCase
| Input
| Handle to a test case. srTEST_CASE_DEFAULT can be used for the default test case.
|-
| szDescr
| Input
| Pointer to a null-terminated string representing the description of test case. Cannot be null.
|}
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;"
| '''Return Value'''
| ''' Description'''
|-
| srOK
| Success
|-
| srERR
| Null string passed in.
|-
| srTEST_ERR_HANDLE_INVALID
| Invalid handle passed in.
|-
| srTEST_WARN_STR_TRUNCATED
| String passed in was too large and was truncated.
|}
<br>
'''Example'''
  #include "srtest.h"
  void tfcase_setDescription(void)
  {
    srTestCaseSetDescription(srTEST_CASE_DEFAULT,
                            "Setting description for default case");
  } <br>
  #ifdef _SCL
  #pragma scl_test_flist(“testfunc”, tfcase_setDescription)
  #endif
 
==== srTestCaseAddComment ====
'''Prototype'''
  srWORD srTestCaseAddComment(srTestCaseHandle_t tTestCase, const srCHAR * szFmtComment, ...)
 
'''Description'''
The srTestCaseAddComment() routine is used to add a comment (aka a log) to be
reported with the specified test case.
 
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;" 
| '''Parameters'''
| '''Type'''
| '''Description'''
|-
| tTestCase
| Input
| Handle to a test case. srTEST_CASE_DEFAULT can be used for the default test case.
|-
| szFmtComment
| Input
| Pointer to a null-terminated string, which can be formatted, representing the comment. Cannot be null.
|-
| ...
| Input (Optional)
| Variable argument list to format the comment szFmtComment.
|}
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;"
| '''Return Value'''
| ''' Description'''
|-
| srOK
| Success
|-
| srERR
| Null string passed in.
|-
| srTEST_ERR_HANDLE_INVALID
| Invalid handle passed in.
|-
| srTEST_WARN_STR_TRUNCATED
| String passed in was too large and was truncated.
|}
<br>
'''Example'''
  #include "srtest.h"
  void tfcase_addComment(void)
  {
    srTestCaseAddComment(srTEST_CASE_DEFAULT,
                        "this comment should print %s", "A STRING");
  }<br>
  #ifdef _SCL
  #pragma scl_test_flist(“testfunc”, tfcase_addComment)
  #endif
 
==== srTestCaseSetStatus ====
'''Prototype'''
  srWORD srTestCaseSetStatus(srTestCaseHandle_t tTestCase, srTestStatus_e eStatus, srDWORD dwDuration)
 
'''Description'''
The srTestCaseSetStatus() routine is used to set the result of test case.
 
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;" 
| '''Parameters'''
| '''Type'''
| '''Description'''
|-
| tTestCase
| Input
| Handle to a test case. srTEST_CASE_DEFAULT can be used for the default test case.
|-
| eStatus
| Input
| Result of the test.
|-
| dwDuration
| Input
| The duration of the test in clock ticks. Using “0” allows the STRIDE Runtime (Intercept Module) to set the time automatically.
|}
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;"
| '''Return Value'''
| ''' Description'''
|-
| srOK
| Success
|-
| srTEST_ERR_HANDLE_INVALID
| Invalid handle passed in.
|}
<br>
'''Example'''
  #include "srtest.h"
  void tfcase_setStatus(void)
  {
    srTestCaseSetStatus(srTEST_CASE_DEFAULT, srTEST_PASS, 0);
  }<br>
  #ifdef _SCL
  #pragma scl_test_flist(“testfunc”, tfcase_setStatus)
  #endif
 
==== srTestCaseSetStatusEx ====
'''Prototype'''
  srWORD srTestCaseSetStatus(srTestCaseHandle_t tTestCase, srTestStatus_e eStatus, srDWORD dwDuration, srLONG lExtendedFailureCode)
 
'''Description'''
The srTestCaseSetStatusEx() routine is used to set the result of test case and allow
specification of an extendedFailureCode.
 
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;" 
| '''Parameters'''
| '''Type'''
| '''Description'''
|-
| tTestCase
| Input
| Handle to a test case. srTEST_CASE_DEFAULT can be used for the default test case.
|-
| eStatus
| Input
| Result of the test. dwDuration Input The duration of the test in clock ticks. Using “0” allows the STRIDE Runtime (Intercept Module) to set the time automatically.
|-
| lExtendedFailureCode
| Input
| The Stride framework uses the extendedFailureCode to capture the numeric results of test method when the test method fails via a numeric (non-void, nonbool) return type.
|}
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;"
| '''Return Value'''
| ''' Description'''
|-
| srOK
| Success
|-
| srTEST_ERR_HANDLE_INVALID
| Invalid handle passed in.
|}
<br>
'''Example'''
  #include "srtest.h"
  void tfcase_setStatusEx(void)
  {
    srTestCaseSetStatusEx(srTEST_CASE_DEFAULT, srTEST_FAIL, 0, -5);
  }<br>
  #ifdef _SCL
  #pragma scl_test_flist(“testfunc”, tfcase_setStatusEx)
  #endif
 
==== srTestSuiteAddAnnotation ====
'''Prototype'''
  srTestAnnotationHandle_t srTestSuiteAddAnnotation(rTestSuiteHandle_t tParent, srTestAnnotationLevel_e eLevel, const srCHAR * szName, const srCHAR * szDescr)
 
'''Description'''
The srTestSuiteAddAnnotation() routine is used to add a new annotation to the specified test suite.
 
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;" 
| '''Parameters'''
| '''Type'''
| '''Description'''
|-
| tParent
| Input
| Handle to the parent test suite to which new annotation is to be added. srTEST_SUITE_DEFAULT can be used for the default test suite.
|-
| eLevel
| Input
| Annotation level. Cannot be empty.
|-
| szName
| Input
| Pointer to a null-terminated string that represents the name of annotation. If null, the default host naming scheme will be used.
|-
| szDescr
| Input
| Pointer to a null-terminated string representing the description of annotation. If null, description will be empty.
|}
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;"
| '''Return Value'''
| ''' Description'''
|-
| srTestAnnotationHandle_t
| HHandle of the newly created annotation. srTEST_ANNOTATION_INVALID indicates failure to create annotation.
|}
<br>
'''Example'''
  #include "srtest.h"
  #include <sstream>
  void tfsuite_addAnnotation(void)
  {
    const std::string prefixName("annotation ");
    const std::string prefixDescr("description of annotation ");
    for (int count = 0; count < 5; ++count)
    {
      std::stringstream strmName;
      std::stringstream strmDescr;
      strmName << prefixName << count;
      strmDescr << prefixDescr << count;
      srTestAnnotationHandle_t annot =
                    srTestSuiteAddAnnotation(srTEST_SUITE_DEFAULT,
                                              srTEST_ANNOTATION_LEVEL_ERROR,
                                              strmName.str().c_str(),
                                              strmDescr.str().c_str());
    }
  }<br>
  #ifdef _SCL
  #pragma scl_test_flist(“testfunc”, tfsuite_addAnnotation)
  #endif
 
==== srTestAnnotationAddComment ====
'''Prototype'''
  srWORD srTestAnnotationAddComment(srTestAnnotationHandle_t tTestAnnotation, const srCHAR * szLabel, const srCHAR * szFmtComment, ...)
 
'''Description'''
The srTestAnnotationAddComment() routine is used to add a comment (aka a log) to be reported with the specified annotation.
 
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;" 
| '''Parameters'''
| '''Type'''
| '''Description'''
|-
| tTestAnnotation
| Input
| Handle to an annotation created using srTestSuiteAddAnnotation.
|-
| szLabel
| Input
| Pointer to a null-terminated string for the label. If null, the default label will be applied.
|-
| szFmtComment
| Input
| Pointer to a null-terminated string, which can be formatted, representing the comment. Cannot be null.
|-
| ...
| Input (Optional)
| Variable argument list to format the comment szFmtComment.
|}
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;"
| '''Return Value'''
| ''' Description'''
|-
| srOK
| Success
|-
| srERR
| Null string passed in.
|-
| srTEST_ERR_HANDLE_INVALID
| Invalid handle passed in.
|-
| srTEST_WARN_STR_TRUNCATED
| String passed in was too large and was truncated.
|}
<br>
'''Example'''
  #include "srtest.h"
  void tfsuiteAnnotation _ addComment(void)
  {
      srTestAnnotationHandle_t annot =
                        srTestSuiteAddAnnotation(srTEST_SUITE_DEFAULT,
                                                  srTEST_ANNOTATION_LEVEL_ERROR,
                                                  “annot”,
                                                  “annot description”);
      srTestAnnotationAddComment(annot,
                                srNULL,
                                "this comment should print %s", "A STRING");
  }<br>
  #ifdef _SCL
  #pragma scl_test_flist(“testfunc”, tfsuiteAnnotation_addComment)
  #endif
 
=== C++ Test Classes ===
The Runtime Test Services APIs work equally well from C test functions and C++ test classes. If, however, you choose to derive your C++ test classes from the STRIDE Runtime base class, ''srTest'', then you will have access to member objects in srTest and their methods that provide the same functionality as the C API. The srTest base class provides two Member Objects, via which you can access functionality:
 
'''''Member Objects''''':
 
*''testSuite'', which has methods:
**'''<code>AddSuite</code>'''
**'''<code>SetName</code>'''
**'''<code>SetDescription</code>'''
**'''<code>AddTest</code>'''
**'''<code>AddAnnotation</code>'''
**'''<code>AddAnnotation.AddComment</code>'''
*''testCase'', which has methods:
**'''<code>SetName</code>'''
**'''<code>SetDescription</code>'''
**'''<code>AddComment</code>'''
**'''<code>SetStatus</code>'''
 
 
''''' Methods of Member Object testSuite '''''
==== AddSuite ====
'''Prototype'''
  srTestSuite AddSuite(const srCHAR * szName = srNULL)
 
'''Description'''
The AddSuite method is used to add a new test suite.
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;" 
| '''Parameters'''
| '''Type'''
| '''Description'''
|-
|szName
| Input (Optional)
| Pointer to null-terminated string that represents the name of test suite. If empty, the default host naming scheme will be used.
|}
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;"
| '''Return Value'''
| ''' Description'''
|-
| srTestSuite
| Newly created test suite class.
|}
<br>
'''Example'''
  #include "srtest.h"
  class srtest_class : public stride::srTest
  {
  public:
    void suiteAddSuite(void);
  };<br>
  void srtest_class::suiteAddSuite(void)
  {
    stride::srTestSuite suite = testSuite.AddSuite("tc Sub Suite");
  } <br>
  #ifdef _SCL
  #pragma scl_test_class(srtest_class)
  #endif
 
==== SetName ====
'''Prototype'''
  srWORD SetName(const srCHAR * szName)
 
'''Description'''
The SetName method is used to set the name of test suite.
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;" 
| '''Parameters'''
| '''Type'''
| '''Description'''
|-
| szName
| Input
| Pointer to null-terminated string representing the name of test suite. Cannot be empty.
|}
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;"
| '''Return Value'''
| ''' Description'''
|-
| srOK
| Success
|-
| srERR
| Null string passed in.
|-
| srTEST_ERR_HANDLE_INVALID
| Invalid handle passed in.
|-
| srTEST_WARN_STR_TRUNCATED
| String passed in was too large and was truncated.
|}
<br>
'''Example'''
  #include "srtest.h"
  class srtest_class : public stride::srTest
  {
  public:
    void suiteSetName(void);
  };<br>
  void srtest_class::suiteSetName(void)
  {
    testSuite.SetName("Setting name for suite");
  }<br>
  #ifdef _SCL
  #pragma scl_test_class(srtest_class)
  #endif
 
==== SetDescription ====
'''Prototype'''
  srWORD SetDescription(const srCHAR * szDescr)
 
'''Description'''
The SetDescription method is used to set the description of test suite.
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;" 
| '''Parameters'''
| '''Type'''
| '''Description'''
|-
| szDescr
| Input
| Pointer to null-terminated string representing the description of test suite. Cannot be empty.
|}
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;"
| '''Return Value'''
| ''' Description'''
|-
| srOK
| Success
|-
| srERR
| Null string passed in.
|-
| srTEST_ERR_HANDLE_INVALID
| Invalid handle passed in.
|-
| srTEST_WARN_STR_TRUNCATED
| String passed in was too large and was truncated.
|}
<br>
'''Example'''
  #include "srtest.h"
  class srtest_class : public stride::srTest
  {
  public:
    void suiteSetDescription(void);
  };<br>
  void srtest_class::suiteSetDescription(void)
  {
    testSuite.SetDescription("Setting description for suite");
  }<br>
  #ifdef _SCL
  #pragma scl_test_class(srtest_class)
  #endif
 
==== AddTest ====
'''Prototype'''
  srTestCase AddTest(const srCHAR * szName = srNULL)
 
'''Description'''
The AddTest method is used to add a new test case to test suite.
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;" 
| '''Parameters'''
| '''Type'''
| '''Description'''
|-
| szName
| Input (Optional)
| Pointer to null-terminated string that represents the name of test case. If empty, the default host naming scheme will be used.
|}
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;"
| '''Return Value'''
| ''' Description'''
|-
| srTestCase
| Newly created test case class.
|}
<br>
'''Example'''
  #include "srtest.h"
  #include <sstream><br>
  class srtest_class : public stride::srTest
  {
  public:
    void suiteAddSuite(void);
  };<br>
  void srtest_class::suiteAddSuite(void)
  {
    const std::string prefix("dynamic test case ");
    for (int count = 0; count < 5; ++count)
    {
      std::stringstream strm;
      strm << prefix << count;
      stride::srTestCase tc = testSuite.AddTest(strm.str().c_str());
      tc.AddComment("this is a dynamic test case");
      tc.SetStatus(srTEST_PASS);
    }
  }<br>
  #ifdef _SCL
  #pragma scl_test_class(srtest_class)
  #endif
 
==== AddAnnotation ====
'''Prototype'''
  srTestAnnoation AddAnnotation(srTestAnnotationLevel_e eLevel, const srCHAR * szName = srNULL, const srCHAR * szDescr = srNULL)
 
 
'''Description'''
The AddAnnotation method is used to add a new annotation to test suite.
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;" 
| '''Parameters'''
| '''Type'''
| '''Description'''
|-
| eLevel
| Input
| Annotation level. Cannot be empty.
|-
| szName
| Input (Optional)
| Pointer to null-terminated string that represents the name of annotation. If empty, the default host naming scheme will be used.
|-
| szDescr
| Input (Optional)
| Pointer to null-terminated string that represents the description of annotation. If empty, the description will be blank.
|}
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;"
| '''Return Value'''
| ''' Description'''
|-
| srTestAnnotation
| Newly created annotation class.
|}
<br>
'''Example'''
  #include "srtest.h"
  #include <sstream><br>
  class srtest_class : public stride::srTest
  {
  public:
      void suiteAddAnnotation(void);
  };<br>
  void srtest_class::suiteAddAnnotation(void)
  {
    const std::string prefixName("annotation ");
    const std::string prefixDescr("description of annotation ");
    for (int count = 0; count < 5; ++count)
    {
      std::stringstream strmName;
      std::stringstream strmDescr;
      strmName << prefixName << count;
      strmDescr << prefixDescr << count;
      stride::srTestAnnotation ta =
              testSuite.AddAnnotation(srTEST_ANNOTATION_LEVEL_INFO,
                                      strmName.str().c_str(),
                                      strmDescr.str().c_str());
    }
  }<br>
  #ifdef _SCL
  #pragma scl_test_class(srtest_class)
  #endif
 
==== AddAnnotation.AddComment ====
'''Prototype'''
  srWORD AddComment(const srCHAR * szLabel, const srCHAR * szFmtComment, ...)
 
'''Description'''
The AddComment method is used to add a comment to an annotation created under a test suite.
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;" 
| '''Parameters'''
| '''Type'''
| '''Description'''
|-
| szLabel
| Input
| Pointer to null-terminated string for the label. If null, the default label will be applied.
|-
| szFmtComment
| Input
| Pointer to null-terminated string, which can be formatted, representing the comment. Cannot be empty.
|-
| ...
| Input (Optional)
| Variable argument list to format the comment szFmtComment.
|}
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;"
| '''Return Value'''
| ''' Description'''
|-
| srOK
| Success
|-
| srERR
| Null string passed in.
|-
| srTEST_ERR_HANDLE_INVALID
| Invalid handle passed in.
|-
| srTEST_WARN_STR_TRUNCATED
| String passed in was too large and was truncated.
|}
<br>
'''Example'''
  #include "srtest.h"<br>
  class srtest_class : public stride::srTest
  {
  public:
    void suiteAnnotationAddComment(void);
  };<br>
  void srtest_class::suiteAnnotationAddComment(void)
  {
    stride::srTestAnnotation ta =
                testSuite.AddAnnotation(srTEST_ANNOTATION_LEVEL_INFO,
                                        “annot”,
                                        “annot description”);
    ta.AddComment("this comment on annotation should print %s", "A STRING");
  }<br>
  #ifdef _SCL
  #pragma scl_test_class(srtest_class)
  #endif
 
 
''''' Methods of Member Object testCase '''''
 
==== SetName ====
'''Prototype'''
  srWORD SetName(const srCHAR * szName)
 
'''Description'''
The SetName method is used to set the name of test case.
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;" 
| '''Parameters'''
| '''Type'''
| '''Description'''
|-
| szName
| Input
| Pointer to null-terminated string representing the name of test case. Cannot be empty.
|}
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;"
| '''Return Value'''
| ''' Description'''
|-
| srOK
| Success
|-
| srERR
| Null string passed in.
|-
| srTEST_ERR_HANDLE_INVALID
| Invalid handle passed in.
|-
| srTEST_WARN_STR_TRUNCATED
| String passed in was too large and was truncated.
|}
<br>
'''Example'''
  #include "srtest.h"<br>
  class srtest_class : public stride::srTest
  {
  public:
  void caseSetName(void);
  };<br>
  void srtest_class::caseSetName(void)
  {
  testCase.SetName("Setting name for case");
  }<br>
  #ifdef _SCL
  #pragma scl_test_class(srtest_class)
  #endif
 
==== SetDescription ====
'''Prototype'''
  srWORD SetDescription(const srCHAR * szDescr)
 
'''Description'''
The SetDescription method is used to set the description of test case.
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;" 
| '''Parameters'''
| '''Type'''
| '''Description'''
|-
| szDescr
| Input
| Pointer to null-terminated string representing the description of test case. Cannot be empty.
|}
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;"
| '''Return Value'''
| ''' Description'''
|-
| srOK
| Success
|-
| srERR
| Null string passed in.
|-
| srTEST_ERR_HANDLE_INVALID
| Invalid handle passed in.
|-
| srTEST_WARN_STR_TRUNCATED
| String passed in was too large and was truncated.
|}
<br>
'''Example'''
  #include "srtest.h"<br>
  class srtest_class : public stride::srTest
  {
  public:
    void caseSetDescription(void);
  };<br>
  void srtest_class::caseSetDescription(void)
  {
    testCase.SetDescription("Setting description for case");
  }<br>
  #ifdef _SCL
  #pragma scl_test_class(srtest_class)
  #endif
 
==== AddComment ====
'''Prototype'''
  srWORD AddComment(const srCHAR * szFmtComment, ...)
 
'''Description'''
The AddComment method is used to add a comment to test case.
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;" 
| '''Parameters'''
| '''Type'''
| '''Description'''
|-
| szFmtComment
| Input
| Pointer to null-terminated string, which can be formatted, representing the comment. Cannot be empty.
|-
| ...
| Input (Optional)
| Variable argument list to format the comment szFmtComment.
|}
|}
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;"
| '''Return Value'''
| ''' Description'''
|-
| srOK
| Success
|-
| srERR
| Null string passed in.
|-
| srTEST_ERR_HANDLE_INVALID
| Invalid handle passed in.
|-
| srTEST_WARN_STR_TRUNCATED
| String passed in was too large and was truncated.
|}
<br>
'''Example'''
  #include "srtest.h"<br>
  class srtest_class : public stride::srTest
  {
  public:
    void caseAddComment(void);
  };<br>
  void srtest_class::caseAddComment(void)
  {
    testCase.AddComment("this comment should print %s", "A STRING");
  }<br>
  #ifdef _SCL
  #pragma scl_test_class(srtest_class)
  #endif
 
==== SetStatus ====
'''Prototype'''
  srWORD SetStatus(srTestStatus_e eStatus, srDWORD dwDuration = 0)
 
'''Description'''
The SetStatus method is used to set the result of the default test case.
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;" 
| '''Parameters'''
| '''Type'''
| '''Description'''
|-
| eStatus
| Input
| Result of the test.
|-
| dwDuration
| Input (Optional)
| The duration of the test in clock ticks. The default is 0.
|}
|}
<br>
{| border="1" cellspacing="0" cellpadding="10" style="align:left;"
| '''Return Value'''
| ''' Description'''
|-
| srOK
| Success
|-
| srTEST_ERR_HANDLE_INVALID
| Invalid handle.
|}
<br>
'''Example'''
  #include "srtest.h"<br>
  class srtest_class : public stride::srTest
  {
  public:
    void caseSetStatus(void);
  };<br>
  void srtest_class::caseSetStatus(void)
  {
    testCase.SetStatus(srTEST_INPROGRESS, 0);
  }<br>
  #ifdef _SCL
  #pragma scl_test_class(srtest_class)
  #endif
 
 
 
Refer to the Reference Guide or the Runtime Developers Guide, both available in the STRIDE Online Help, for detailed information about any of these functions.
 
== Scripting a Test Unit ==
 
To automate the execution and reporting of a Test Unit a script is required. Scripts can be written by hand or automatically generated using the Script Wizard and a corresponding template script. A scripting tool for executing a test unit is the [[AutoScript#ascript.TestUnits|AutoScript TestUnits]] collection. An [[AutoScript#ascript.TestUnits.Item|Ascript TestUnit]] object assembles all of the reporting information for the test unit and its corresponding test methods.
 
*Require useage of the [[AutoScript#ascript.TestUnits|AutoScript TestUnits]] collection
*Can be written by hand (refer below)
*Can leverage [[Templates|Templates]] via the Script Wizard
*Order of multiple test units dictated by SUID assignment
 
<br>
 
=== JScript example for a single test unit  ===
 
The following example script is used to harness a test unit that has been captured using #pragma scl_test_class(Simple).
 
  // Ensure test unit exists
  if (ascript.TestUnits.Item("Simple")&nbsp;!= null )
    ascript.TestUnits.Item("Simple").Run();
 
=== Perl example for a single test unit  ===
 
The following example script is used to harness a test unit that has been captured using #pragma scl_test_class(Simple).
 
<br>
 
  use strict;
  use Win32::OLE;
  Win32::OLE-&gt;Option(Warn =&gt; 3);
 
  my $tu = $main::ascript-&gt;TestUnits-&gt;Item("Simple");
  if (defined $tu) {
    $tu-&gt;Run();
  }
 
=== JScript example for multiple test units  ===
 
The following example script is used to harness two test units that have been captured using #pragma scl_test_class(Simple1) and #pragma scl_test_class(Simple2).
 
<br>
 
  var Units = ["Simple1","Simple2"];
 
  // iterate through each function
  for (i in Units)
  {
    var tu = ascript.TestUnits.Item(Units[i]);
    if ( tu&nbsp;!= null )
      tu.Run();
  }
 
=== Perl example for multiple test units  ===
 
The following example script is used to harness two test units that have been captured using #pragma scl_test_class(Simple1) and #pragma scl_test_class(Simple2).
 
<br>
 
  use strict;
  use Win32::OLE;
  Win32::OLE-&gt;Option(Warn =&gt; 3);
 
  # initialize an array with all selected function names
  my @UnitNames = ("Simple1","Simple2");
  foreach (@UnitNames) { 
    my $tu = $main::ascript-&gt;TestUnits-&gt;Item($_-&gt;[1]);
    die "TestUnit not found: $_-&gt;[1]\n" unless (defined $tu);
    $tu-&gt;Run();
  }
 
[[Category:Reference]]

Latest revision as of 23:09, 25 September 2013

What are STRIDE Test Units?

STRIDE Test Units is a general term for xUnit-style test modules running within the STRIDE runtime framework. These tests--written in C and C++--are compiled and linked with your embedded software and run in-place on your target hardware. They are suitable for both developer unit testing as well as end-to-end integration testing.

An external Test Runner is provided which controls the execution of the tests and publishes test results to the local file system and optionally to S2's Internet STRIDE Test Space.

Test Unit Features

In all cases, STRIDE Test Units provide the following capabilities typical of all xUnit-style testing frameworks:

  • Specification of a test as a test method
  • Aggregation of individual tests into test suites which form execution and reporting units
  • Specification of expected results within test methods (typically by using one or more Test Code Macros)
  • Test fixturing (optional setup and teardown)
  • Test parametrization (optional constructor/initialization parameters)
  • Automated execution
  • Automated results report generation

Unique STRIDE Test Unit Features

In addition, STRIDE Test Units offer these unique features:

On-Target Execution
Tests execute on the target hardware in a true operational environment. Execution and reporting is controlled from a remote desktop (Windows, Linux or FreeBSD) host
Dynamic Test and Suite Generation
Test cases and suites can be created and manipulated at runtime
Test Doubles
Dynamic runtime function substitution to implement on-the-fly mocks, stubs, and doubles
Behavior Testing (Test Points)
Support for testing of asynchronous activities occurring in multiple threads
Multiprocess Testing Framework
Support for testing across multiple processes running simultaneously on the target
Automatic Timing Data Collection
Duration are automatically measured for each test case.
Automatic Results Publishing to Local Disk and Internet
Automatic publishing of test results to STRIDE Test Space
Retrieved from "https://www.stridewiki.com/index.php?title=Test_Units_Overview&oldid=13977"