|
|
| (305 intermediate revisions by 9 users not shown) |
| Line 1: |
Line 1: |
| ==Introduction== | | == What are STRIDE Test Units? == |
|
| |
|
| STRIDE enables testing of C++ code through the use of xUnit-style test classes. Test classes can be written by engineers, captured using an SCL pragma, and executed from the host. STRIDE facilitates the execution of some or all of the test classes by automatically creating entry points for the execution of test classes on the target. | | '''STRIDE Test Units''' is a general term for [http://en.wikipedia.org/wiki/XUnit xUnit-style] test modules running within the STRIDE runtime framework. These tests--written in C and C++--are compiled and linked with your embedded software and run in-place on your target hardware. They are suitable for both developer unit testing as well as end-to-end integration testing. |
|
| |
|
| ==Using C++ test classes==
| | An external [[Stride Runner|Test Runner]] is provided which controls the execution of the tests and publishes test results to the local file system and optionally to S2's Internet [[STRIDE Test Space]]. |
|
| |
|
| ===Prerequisites=== | | == Test Unit Features == |
| | In all cases, STRIDE Test Units provide the following capabilities typical of all [http://en.wikipedia.org/wiki/XUnit xUnit-style] testing frameworks: |
| | * Specification of a test as a test method |
| | * Aggregation of individual tests into test suites which form execution and reporting units |
| | * Specification of expected results within test methods (typically by using one or more [[Test Code Macros]]) |
| | * Test fixturing (optional setup and teardown) |
| | * Test parametrization (optional constructor/initialization parameters) |
| | * Automated execution |
| | * Automated results report generation |
|
| |
|
| The C++ test class support described here only works in STRIDE version 2.1.0000 or later. The host PC must also have a recent distribution of [http://www.activestate.com/Products/activeperl/ ActiveState Perl installed.]
| | === Unique STRIDE Test Unit Features === |
| | In addition, STRIDE Test Units offer these unique features: |
| | ; On-Target Execution |
| | : Tests execute on the target hardware in a true operational environment. Execution and reporting is controlled from a remote desktop (Windows, Linux or FreeBSD) host |
| | ; Dynamic Test and Suite Generation |
| | : Test cases and suites can be created and manipulated at runtime |
| | ; [[Using Test Doubles|Test Doubles]] |
| | : Dynamic runtime function substitution to implement on-the-fly mocks, stubs, and doubles |
| | ; [[Test Point Testing in C/C++|Behavior Testing]] (Test Points) |
| | : Support for testing of asynchronous activities occurring in multiple threads |
| | ; Multiprocess Testing Framework |
| | : Support for testing across multiple processes running simultaneously on the target |
| | ; Automatic Timing Data Collection |
| | : Duration are automatically measured for each test case. |
| | ; Automatic Results Publishing to Local Disk and Internet |
| | : Automatic publishing of test results to [[STRIDE Test Space]] |
|
| |
|
| ===How to get started===
| | [[Category:Test Units]] |
| | |
| The required steps to get started with writing C++ test classes are outlined as follows:
| |
| | |
| <ol>
| |
| <li>Create a new Studio workspace (or open an existing one).</li>
| |
| <li>Set the workspace to compile in C++ mode (Tools->Settings->Compile as Cpp).</li>
| |
| <li>Add '''%STRIDE_DIR%'''\inc\srtest.h to the Source Files folder of your workspace.</li>
| |
| <li>Set '''%STRIDE_DIR%'''\scripts\TestClasses\TestClass_Preprocess.pl (or .js) to your Workspace General Pre-Compile Script property (Highlight "Workspace" and look in the Properties View).
| |
| :This script is a preprocessor and code generator. It searches all source files in the current workspace and looks for test classes that have been captured via the ''scl_test_class'' pragma. It then generates c-linkage wrapper functions that instantiate and execute each test class. By default, this generated source file has a name of the form <workspace_name>_run.cpp and will be located in the workspace directory. If you need to customize the name and/or location of this generated file, simply create your own version of the preprocessor script and set the '''outFile''' property of the '''''STRIDE.testclass_codegen''''' object prior to calling the generate method. Note: This preprocessing script must always '''preceed''' the compilation step.</li>
| |
| <li>Set a script to generate the Intercept Module(IM) '''after''' the compilation step.
| |
| :For the simple STUB generation required for C++ test class execution, you can use '''%STRIDE_DIR%'''\scripts\TestClasses\TestClass_IMCreate.pl (or .js) for this purpose. Set the script to your Workspace General Post-Compile Script property (Also in "Workspace" Properties). If you need to customize the name and/or location of the generated Intercept Module simply create your own version of this script.</li>
| |
| <li>Add scripts to build and execute your application. If you are using a host based simulator, we have examples available of both. If you are using actual devices, the steps required for building and starting the application are specific to the target environment.</li>
| |
| <li>Create one or more test classes to implement your C++ test logic. See below for more information on creating test classes.</li>
| |
| <li>Ensure that the Studio workspace include path has the location to all of your test class declaration (header) files.</li>
| |
| <li>Once you have created one or more test classes, save the workspace and simply compile the workspace.</li>
| |
| <li>[''Optional''] If your application is running, you can test execute individual test classes interactively using the Studio interface view. Open the user interface view corresponding to the test class you would like to execute and call it. The return values will indicate how many tests produced each of 4 result types. Furthermore, the input to the entry point will allow you to select all methods for execution (the default) or individual methods -- this is done via a dropdown list of enumerated values.</li>
| |
| :* Once you are confident that the test classes are behaving as expected, you can generate one or more execution scripts using the Script Wizard. Sample templates for executing test class entry points are provided in the '''%STRIDE_DIR%'''\templates\Script Wizard directory.
| |
| :* For integration with larger regression test workspaces, we recommend that engineers check-in their test class code and, optionally, the template generated scripts that can be used to execute their test classes. </ol>
| |
| | |
| ===Pragmas for Test Classes===
| |
| STRIDE supports three pragmas for capturing and qualifying test classes:
| |
| | |
| * '''<code>scl_test_class ( class )</code>''': Declares a test class as captured. Once captured, STRIDE will generate the appropriate code for executing the test methods in the class.
| |
| * '''<code>scl_test_setup ( class , method )</code>''': [optional] Declares a member method to be a setup fixture for the class. If specified, the setup method will be called before the execution of each test method.
| |
| * '''<code>scl_test_teardown ( class , method )</code>''': [optional] Declares a member method to be a teardown fixture for the class. If specified, the teardown method will be called after the execution of each test method.
| |
| | |
| ==C++ test class requirements==
| |
| | |
| We support several variations on typical xUnit-style test classes. The additional features we support include:
| |
| * Test status can be set using STRIDE Runtime APIs ''or'' by specifying simple return types for test methods.
| |
| * Test writers can create additional child suites and tests at runtime by using Runtime APIs.
| |
| * We do not rely on exceptions for reporting of status.
| |
| | |
| The STRIDE test class framework has the following requirements of each test class:
| |
| * The test class must have a suitable default (no-argument) constructor.
| |
| * The test class must have one or more public methods suitable as test methods. Allowable test methods always take no arguments (void) and return either void or simple integer types (int, short, long, char or bool). At this time, we do not allow typedef types or macros for the return values specification.
| |
| * the scl_test_class pragma must be applied to the class.
| |
| | |
| ===Simple example using return values for status===
| |
| | |
| #include <srtest.h>
| |
|
| |
| class Simple {
| |
| public:
| |
| int tc_Int_ExpectPass(void) {return 0;}
| |
| int tc_Int_ExpectFail(void) {return -1;}
| |
| bool tc_Bool_ExpectPass(void) {return true;}
| |
| bool tc_Bool_ExpectFail(void) {return false;}
| |
| };
| |
| #ifdef _SCL
| |
| #pragma scl_test_class(Simple)
| |
| #endif
| |
| | |
| ===Simple example using runtime test service APIs===
| |
| | |
| #include <srtest.h>
| |
|
| |
| class RuntimeServices_basic {
| |
| public:
| |
| void tc_ExpectPass(void)
| |
| {
| |
| srTestCaseAddComment(srTEST_CASE_DEFAULT, "this test should pass");
| |
| srTestCaseSetStatus(srTEST_CASE_DEFAULT, srTEST_PASS, 0);
| |
| }
| |
| void tc_ExpectFail(void)
| |
| {
| |
| srTestCaseAddComment(srTEST_CASE_DEFAULT, "this test should fail");
| |
| srTestCaseSetStatus(srTEST_CASE_DEFAULT, srTEST_FAIL, 0);
| |
| }
| |
| void tc_ExpectInProgress(void)
| |
| {
| |
| srTestCaseAddComment(srTEST_CASE_DEFAULT, "this test should be in progress");
| |
| }
| |
| };
| |
| #ifdef _SCL
| |
| #pragma scl_test_class(RuntimeServices_basic)
| |
| #endif
| |
| | |
| ===Simple example using srTest base class===
| |
| | |
| #include <srtest.h>
| |
|
| |
| class MyTest : public stride::srTest {
| |
| public:
| |
| void tc_ExpectPass(void)
| |
| {
| |
| testCase.AddComment("this test should pass");
| |
| testCase.SetStatus(srTEST_PASS, 0);
| |
| }
| |
| void tc_ExpectFail(void)
| |
| {
| |
| testCase.AddComment("this test should fail");
| |
| testCase.SetStatus(srTEST_FAIL, 0);
| |
| }
| |
| void tc_ExpectInProgress(void)
| |
| {
| |
| testCase.AddComment("this test should be in progress");
| |
| }
| |
| int tc_ChangeMyName(void)
| |
| {
| |
| testCase.AddComment("this test should have name = MyChangedName");
| |
| testCase.SetName("MyChangedName");
| |
| return 0;
| |
| }
| |
| int tc_ChangeMyDescription(void)
| |
| {
| |
| testCase.AddComment("this test should have a description set");
| |
| testCase.SetDescription("this is my new description");
| |
| return 0;
| |
| }
| |
| };
| |
| #ifdef _SCL
| |
| #pragma scl_test_class(MyTest)
| |
| #endif
| |
| | |
| ==Runtime Test Services==
| |
| | |
| The Runtime Test Services are a set of APIs in the STRIDE Runtime that facilitate the writing of target based test code. These APIs make up an optional portion of the STRIDE Runtime and can be used to communicate additional information about tests to the host based reporting mechanism. These APIs also allow target test code to create additional test suites and test cases dynamically at runtime.
| |
| | |
| The following C APIs are provided:
| |
| | |
| * '''<code>srTestSuiteAddSuite</code>''': creates an additional sub-suite at runtime.
| |
| * '''<code>srTestSuiteSetName</code>''': sets the name of the specified suite.
| |
| * '''<code>srTestSuiteSetDescription</code>''': sets the description of the specified suite.
| |
| * '''<code>srTestSuiteAddTest</code>''': creates an additional test case at runtime.
| |
| * '''<code>srTestCaseSetName</code>''': sets the name of the specified test case.
| |
| * '''<code>srTestCaseSetDescription</code>''': sets the description of the specified test case.
| |
| * '''<code>srTestCaseAddComment</code>''': adds a comment to the specified test case.
| |
| * '''<code>srTestCaseSetStatus</code>''': explicitly sets the status for the specified test case.
| |
| | |
| These C APIs work equally well from C test functions and C++ test classes. If, however, you choose to derive your C++ test classes from the STRIDE Runtime base class, ''srTest'', then you will have access to member objects in srTest and their methods that provide the same functionality as the C API. The srTest base class provides two Member Objects, via which you can access functionality:
| |
| | |
| # '''''Member Objects''''':
| |
| #* ''testSuite'', which has methods:
| |
| #** '''<code>AddSuite</code>'''
| |
| #** '''<code>SetName</code>'''
| |
| #** '''<code>SetDescription</code>'''
| |
| #** '''<code>AddTest</code>'''
| |
| #* ''testCase'', which has methods:
| |
| #** '''<code>SetName</code>'''
| |
| #** '''<code>SetDescription</code>'''
| |
| #** '''<code>AddComment</code>'''
| |
| #** '''<code>SetStatus</code>'''
| |
| | |
| See the STRIDE Reference guide in On-Line-Help or the Runtime Developers Guide for detailed information about any of these functions.
| |
| | |
| ==Libraries/Components==
| |
| | |
| There are several installed components that provide some of the core functionality for test class harnessing and execution. These components are installed and registered with the core product installation.
| |
| | |
| ===STRIDE.testclass===
| |
| | |
| The '''''STRIDE.testclass''''' component is used for executing test classes. It handles the underlying communication with the runtime and creates results in a specified reporter suite. This object supports the following properties and methods:
| |
| * ''ascript'' : Input only. This is the ascript instance for the current execution environment. This is '''required''' prior to calling the '''run''' method. In most cases, you can simply pass the global ascript instance from your script that executes in your workspace.
| |
| * ''parentSuite'' : Input only. This is a reporter suite object into which results for the test class should be added. A sub-suite corresponding to the test class will be created in the given parent suite and all test method results will be added to this new child suite. This is '''required''' prior to calling the '''run''' method. This suite will also serve as the default suite during execution, into which dynamic entities will be placed when srTEST_SUITE_DEFAULT is specified.
| |
| * ''dataFile'' : Input only. This specifies the name/location of the test class meta-data file (generated by the preprocessing step). This is '''required''' prior to calling the '''run''' method.
| |
| * '''run''' : Method. This initiates the running of the specified test class object. This method requires a single argument -- the name of the test class entry point function that was generated by the preprocessing step.
| |
| | |
| ===STRIDE.testclass_codegen===
| |
| | |
| The TestClass_Preprocess.pl/js script demonstrates how to invoke the functionality exposed in the '''''STRIDE.testclass_codegen''''' object. This object exposes the following properties and methods:
| |
| * ''exceptions'' : Input only. Indicates whether or not to generate exception handling code in the test class entry point functions. If you do not have exceptions enabled on your target, set this to 0 -- otherwise we recommend you set it to 1 to enable exception handling code in the test class entry point.
| |
| * ''outFile'' : Input only. The name/location of the generated file. This will default to a file named <workspace_name>_run.cpp (where {WORKSPACE} is the name of the current workspace) and will be located in the workspace directory.
| |
| * ''dataFile'' : Input only. The name/location of the meta-data file generated by the preprocessor. This meta-data is used by the STRIDE.testclass library to determine the name of test classes and methods as they execute. The default name for this file is <workspace_name>.tcrd, located in the workspace directory. If you override this property, be sure to override the same property in '''''STRIDE.testclass''''' wherever it is called.
| |
| * ''studio'' : Input only. This is the automation object interface to the current execution environment. In most cases, you can simply pass the current global studio object from the script in your workspace. This is '''required''' prior to calling '''generate'''.
| |
| * '''generate''' : Method. Call this method when you are ready to generate the test class entry points.
| |
| | |
| ==Script Templates==
| |
| | |
| There are several installed script templates for use with the STRIDE Studio Script Wizard functionality. These templates provide a way to invoke the '''STRIDE.testclass''' object for particular test class entry point functions.
| |
| | |
| There are two variants of the the template scripts for test classes. The ''TestClass_Multiple'' templates are intended for use in running multiple test class entry point functions with a single script. The set of functions is stored in an array (generated by the Script Wizard) and execution merely iterates over the set. The ''TestClass_Single'' templates can be used for creating a script to execute a single test class entry point.
| |
| | |
| ==Sample Workspace==
| |
| | |
| A sample workspace called ''TargetTestCode'' is installed with the STRIDE samples (an optional section when installing). This workspace not only includes several examples of test classes and functions, but also demonstrates several operational aspects of target based verification. In particular, it demonstrates:
| |
| * ''How to call the preprocessor script before compilation by specifying the Pre-compile workspace script''. This is a convenient way to ensure that your test classes are properly captured before compiling your source code.
| |
| * ''How to use a host based simulation application to run your test code''. This allows the test developers to execute their tests in a simulation environment -- which is typically easier to build and execute -- before building and running their real target device. We provide a sample host application with this workspace.
| |
| * ''How to execute test classes and test functions from scripts''. Two scripts are provided, one responsible for executing the test classes, the other handles the test functions execution. Both scripts were generated using the Script Wizard, using the variation that executes multiple interfaces in a single script.
| |
| * ''How to use several different styles of test classes and test functions within the STRIDE framework''. We provide sample test classes that show to to use the srTest base class and how to use no base class and simply call the runtime test service APIs directly. We also demonstrate how to use the runtime test services from c-based test functions.
| |
| | |
| '''NOTE:''' This sample requires a path to std c/c++ header files. The workspace is set to use the default include location for MS Visual Studio 2003. Users will need to change the include path for the TargetTestCode workspace to point to a different location if you are using a different version of visual studio.
| |
| | |
| [[Category:Test Utilities]] | |