|
|
| Line 1: |
Line 1: |
| | ==Tests Description== |
|
| |
|
| ==Sample Test Classes==
| | ''NOTE:'' each of the example test classes is grouped in namespace corresponding to its category (e.g. ''Basic'' or ''Runtime Services''). This is something shown for organizational purposes only -- it is '''not''' a general requirement that test classes be placed into namespaces. |
|
| |
|
| Once you have built the sample application and executed the test classes it contains, you can take time to peruse the test class source and the corresponding results that each produces. This section provides a brief description for each.
| | ===Basic=== |
|
| |
|
| ''NOTE:'' each of the example test classes is grouped in namespace corresponding to its top-level category (e.g. ''Basic'' or ''Runtime Services''). This is something shown for organizational purposes only -- it is '''not''' a general requirement that test classes be placed into namespaces.
| | These examples cover the simplest, easiest way to code a STRIDE test class. These examples use simple [http://en.wikipedia.org/wiki/Plain_Old_Data_Structures POD] return types to indicate status. |
|
| |
|
| ===Basic Examples=== | | ====Simple==== |
| | |
| These examples cover the simplest, easiest way to code a STRIDE test class. These examples use simple [http://en.wikipedia.org/wiki/Plain_Old_Data_Structures POD] return types to indicate status and do not annotate the tests with any rich information (such as comments).
| |
| | |
| ====01_01_Basic_Simple====
| |
|
| |
|
| This example demonstrates passing and failing tests using the four primary POD types that infer status from (int, bool, short, and char). | | This example demonstrates passing and failing tests using the four primary POD types that infer status from (int, bool, short, and char). |
|
| |
|
| ====01_02_Basic_Fixtures==== | | ====Fixtures==== |
| | |
| This example demonstrates how to use [[Test_Units#Pragmas_for_Test_Units|setup and teardown]] fixtures. The setup and teardown methods are called immediately before and after the execution of each test method, respectively.
| |
| | |
| ====01_03_Basic_Exceptions====
| |
|
| |
|
| This example demonstrates how exceptions thrown from a test method are caught by the intercept module and noted in the results. Any exception that is caught by the harness is assumed to indicate failure. | | This example demonstrates how to use [[Test_Unit_Pragmas#Fixturing_Pragmas|setup and teardown]] fixtures. The setup and teardown methods are called immediately before and after the execution of each test method, respectively. |
|
| |
|
| ====01_04_Basic_Constructors==== | | ====Exceptions==== |
|
| |
|
| This example demonstrates how test classes may have non-trivial constructors. These arguments can be passed using the [[AutoScript#ascript.TestUnits|ascript]] scripting model for test units. | | This example demonstrates how exceptions thrown from a test method are caught by the intercept module and noted in the results. Any exception that is caught by the harness is assumed to indicate failure. If you want to write tests for expected exceptions, consider using our [[Pass/Fail_Macros#Exception_Macros exception macros]] |
|
| |
|
| ===Runtime Services Examples=== | | ===Runtime Services=== |
|
| |
|
| These examples cover basic usage of the Runtime Test Services API (as declared in srtest.h). | | These examples cover basic usage of the Runtime Test Services API (as declared in srtest.h). |
|
| |
|
| ====02_01_RuntimeServices_Simple==== | | ====Simple==== |
|
| |
|
| This example demonstrates how to use [[Test_Units#srTestCaseSetStatus|srTestCaseSetStatus]] to set status, [[Test_Units#srTestCaseAddComment|srTestCaseAddComment]] to add a comment, and srTEST_ADD_COMMENT_WITH_LOCATION to add a comment that automatically includes line and file information. | | This example demonstrates how to use [[Test_Units#srTestCaseSetStatus|srTestCaseSetStatus]] to set status, [[Test_Units#srTestCaseAddComment|srTestCaseAddComment]] to add a comment, and srTEST_ADD_COMMENT_WITH_LOCATION to add a comment that automatically includes line and file information. |
|
| |
|
| ====02_02_RuntimeServices_Dynamic==== | | ====Dynamic==== |
|
| |
|
| This example demonstrates how to use [[Test_Units#srTestSuiteAddSuite|srTestSuiteAddSuite]], [[Test_Units#srTestSuiteAddCase|srTestSuiteAddCase]], [[Test_Units#srTestSuiteAddAnnotation|srTestSuiteAddAnnotation]], and [[Test_Units#srTestAnnotationAddComment|srTestAnnotationAddComment]] for dynamic suite, test, and annotation creation in the context of a single test method. | | This example demonstrates how to use [[Test_Units#srTestSuiteAddSuite|srTestSuiteAddSuite]], [[Test_Units#srTestSuiteAddCase|srTestSuiteAddCase]], [[Test_Units#srTestSuiteAddAnnotation|srTestSuiteAddAnnotation]], and [[Test_Units#srTestAnnotationAddComment|srTestAnnotationAddComment]] for dynamic suite, test, and annotation creation in the context of a single test method. |
|
| |
|
| ====02_03_RuntimeServices_Override==== | | ====Override==== |
|
| |
|
| This example demonstrates how to use [[Test_Units#srTestCaseSetStatus|srTestCaseSetStatus]] to override the status that would otherwise be inferred from the return value. | | This example demonstrates how to use [[Test_Units#srTestCaseSetStatus|srTestCaseSetStatus]] to override the status that would otherwise be inferred from the return value. |
|
| |
|
| ====02_04_RuntimeServices_VarComment==== | | ====VarComment==== |
|
| |
|
| This example demonstrates the use of [http://en.wikipedia.org/wiki/Printf printf] style format strings with [[Test_Units#srTestCaseAddComment|srTestCaseAddComment]]. | | This example demonstrates the use of [http://en.wikipedia.org/wiki/Printf printf] style format strings with [[Test_Units#srTestCaseAddComment|srTestCaseAddComment]]. |
|
| |
|
| ===srTest Examples=== | | ===srTest=== |
|
| |
|
| These examples show to how to use the [[Test_Units#C.2B.2B_Test_Classes|stride::srTest]] base class for your test classes. When you publicly inherit from srTest, you get access to default testCase and testSuite members and their associated methods. | | These examples show to how to use the [[Test_Units#C.2B.2B_Test_Classes|stride::srTest]] base class for your test classes. When you publicly inherit from srTest, you get access to default testCase and testSuite members and their associated methods. |
|
| |
|
| ====03_01_srTest_Simple==== | | ====Simple==== |
|
| |
|
| This example demonstrates the use of [[Test_Units#SetStatus|testCase.setStatus]] to set the status for test cases. | | This example demonstrates the use of [[Test_Units#SetStatus|testCase.setStatus]] to set the status for test cases. |
|
| |
|
| ====03_02_srTest_Dynamic==== | | ====Dynamic==== |
|
| |
|
| This example demonstrates how to use [[Test_Units#AddSuite|testSuite.AddSuite]], [[Test_Units#AddTest|testSuite.AddTest]], [[Test_Units#AddAnnotation|testSuite.AddAnnotation]], and [[Test_Units#AddAnnotation.AddComment|testSuite.AddAnnotation.AddComment]] for dynamic suite, test case, and annotation creation within the context of one test method. | | This example demonstrates how to use [[Test_Units#AddSuite|testSuite.AddSuite]], [[Test_Units#AddTest|testSuite.AddTest]], [[Test_Units#AddAnnotation|testSuite.AddAnnotation]], and [[Test_Units#AddAnnotation.AddComment|testSuite.AddAnnotation.AddComment]] for dynamic suite, test case, and annotation creation within the context of one test method. |
|
| |
|
| ==Test Class Execution==
| |
|
| |
| This sample demonstrates two different techniques for executing test classes.
| |
|
| |
| ===Command Line Execution===
| |
|
| |
| Command line execution for test classes is done using the [[Test_Runners#TestUnitRun.pl|TestUnitRun utility]]. Here are several examples of specific syntax to execute test classes. All of these commands can be invoked from a standard [http://en.wikipedia.org/wiki/Command_Prompt_(Windows) command shell] (or other shell of your choosing) and the arguments shown assume that the commands are executed with the sample's directory as the starting directory. You must have your TestClass.exe application running in order for the runner to be able to initiate a connection to the target simulator. In addition, you should verify that your %STRIDE_DIR%\bin\transport.cfg file is using the TCP transport to connect to port 8000 (these are the default settings when the product is installed).
| |
|
| |
| ====Simple execution of all test units====
| |
|
| |
| The following command executes all of the test units found in the STRIDE database you have previously generated. For the purpose of this sample, since there is only one database, the -d parameter is not strictly needed, but it is shown here for completeness.
| |
|
| |
| TestUnitRun.pl -d TestClass.sidb
| |
|
| |
| This command executes all Test Units found in the database in descending alpha-numeric sort order. Any Test Class initialization arguments are given default values (typically zero or NULL).
| |
|
| |
| When you run this command, you should see console output like:
| |
|
| |
| Attempting connection using [Sockets (S2)] transport ...
| |
| Connected to device.
| |
| Initializing STRIDE database objects...
| |
| Done.
| |
| Running Test Basic::Constructors...
| |
| Running Test Basic::Exceptions...
| |
| Running Test Basic::Fixtures...
| |
| Running Test Basic::Simple...
| |
| Running Test RuntimeServices::Dynamic...
| |
| Running Test RuntimeServices::Override...
| |
| Running Test RuntimeServices::Simple...
| |
| Running Test RuntimeServices::VarComment...
| |
| Running Test srTest::Dynamic...
| |
| Running Test srTest::Simple...
| |
| Disconnected from device.
| |
| Test Results saved to C:\STRIDE\Samples\TestUnits\TestClass\TestClass.xml
| |
| Test Report saved to C:\STRIDE\Samples\TestUnits\TestClass\TestClass.html
| |
| ***************************************************************************
| |
| Results Summary
| |
| ***************************************************************************
| |
| Passed: 26
| |
| Failed: 12
| |
| In Progress: 2
| |
| Not Applicable: 2
| |
| ...in 12 suites.
| |
| ***************************************************************************
| |
|
| |
| ====Execution based on an order file====
| |
|
| |
| TestUnitRun can optionally base its execution on simple text file input. A simple order file, ''RunSimple.txt'', is provided which specifies a subset of all the Test Classes in this sample. This order file also shows how to create subsuites in the final output by using the special '''{suitepath}''' syntax, as described in [[Test_Runners#Usage|the usage section]].
| |
|
| |
| TestUnitRun.pl -d TestClass.sidb -o RunSimple.txt
| |
|
| |
| ...and will produce this output:
| |
|
| |
| Attempting connection using [Sockets (S2)] transport ...
| |
| Connected to device.
| |
| Initializing STRIDE database objects...
| |
| Done.
| |
| Running Test Basic::Simple...
| |
| Running Test RuntimeServices::Simple...
| |
| Running Test srTest::Simple...
| |
| Disconnected from device.
| |
| Test Results saved to C:\STRIDE\Samples\TestUnits\TestClass\TestClass.xml
| |
| Test Report saved to C:\STRIDE\Samples\TestUnits\TestClass\TestClass.html
| |
| ***************************************************************************
| |
| Results Summary
| |
| ***************************************************************************
| |
| Passed: 13
| |
| Failed: 6
| |
| In Progress: 2
| |
| Not Applicable: 2
| |
| ...in 6 suites.
| |
| ***************************************************************************
| |
|
| |
| ====Execution based on filesystem order====
| |
|
| |
| TestUnitRun can also try to infer execution order and suite hierarchy from filesystem organization. If you have organized your test class source files (only the files that contain the scl_test_class pragma matter here) in a filesystem hierarchy that you want to mimic in your tests, you can specify a root of a directory tree that contains the relevant test class source. TestUnitRun will walk the directory structure and determine order and hierarchy of tests based on the directory structure. To see an example of this in action, you can execute this command with the sample:
| |
|
| |
| TestUnitRun.pl -d TestClass.sidb -f Tests
| |
|
| |
| This will cause the runner to examine the Tests subdirectory structure and build a hierarchy of tests based on that directory tree. Subdirectories will map to suites in the final report. Here is the output for this example:
| |
|
| |
| Attempting connection using [Sockets (S2)] transport ...
| |
| Connected to device.
| |
| Initializing STRIDE database objects...
| |
| Done.
| |
| Running Test Basic::Simple...
| |
| Running Test Basic::Fixtures...
| |
| Running Test Basic::Exceptions...
| |
| Running Test Basic::Constructors...
| |
| Running Test RuntimeServices::Simple...
| |
| Running Test RuntimeServices::Dynamic...
| |
| Running Test RuntimeServices::Override...
| |
| Running Test RuntimeServices::VarComment...
| |
| Running Test srTest::Simple...
| |
| Running Test srTest::Dynamic...
| |
| Disconnected from device.
| |
| Test Results saved to C:\STRIDE\Samples\TestUnits\TestClass\TestClass.xml
| |
| Test Report saved to C:\STRIDE\Samples\TestUnits\TestClass\TestClass.html
| |
| ***************************************************************************
| |
| Results Summary
| |
| ***************************************************************************
| |
| Passed: 26
| |
| Failed: 12
| |
| In Progress: 2
| |
| Not Applicable: 2
| |
| ...in 15 suites.
| |
| ***************************************************************************
| |
|
| |
| ===Workspace-based Execution===
| |
|
| |
| TestClass.ssw, a workspace in the TestClass directory, demonstrates the use of script execution with Studio to manage test order and hierarchy. This workspace was created using [[WorkspaceSetup.pl]]. The setup and teardown folders provide basic infrastructure scripts that start/stop the simulator application (TestClass.exe) and to manage traceviews used for [[Runtime_Reference#Logging_Services|srPrint]] message collection. The scripts that drive the testing are in the workspace '''test''' folder. What follows is a brief description for each.
| |
|
| |
| ====RunAll====
| |
|
| |
| This folder contains a script, All.js, that iterates through the entire collection of test units and executes them one at a time. The order of execution will be in ascending alpabetical order (by name) since the [[AutoScript#ascript.TestUnits|ArrangeBy]] collection method was called.
| |
|
| |
| ====CallConstructor====
| |
|
| |
| This folder contains a simple script example that shows how to invoke test classes with constructor arguments. In this example, the Basic::Constructors test class is executed twice with different initialization (constructor) arguments both times.
| |
|
| |
| ====Run Individual====
| |
|
| |
| This folder shows how to use individual scripts to execute test classes. Each script has the following form:
| |
|
| |
| ascript.TestUnits.Item('''TEST_CLASS_NAME''').Run();
| |
|
| |
| The '''TEST_Class_NAME''' is the name of the scl_test_class test to be run. The order and hierarchy of each item may be changed via the Studio tree control by moving the item within the scripts and/or folders. The sample contains individual scripts for a few of the sample scl_test_class tests - you are free to move, add, or delete any items as you experiment with the workspace.
| |
|
| |
|
| [[Category: Samples]] | | [[Category: Samples]] |
Tests Description
NOTE: each of the example test classes is grouped in namespace corresponding to its category (e.g. Basic or Runtime Services). This is something shown for organizational purposes only -- it is not a general requirement that test classes be placed into namespaces.
Basic
These examples cover the simplest, easiest way to code a STRIDE test class. These examples use simple POD return types to indicate status.
Simple
This example demonstrates passing and failing tests using the four primary POD types that infer status from (int, bool, short, and char).
Fixtures
This example demonstrates how to use setup and teardown fixtures. The setup and teardown methods are called immediately before and after the execution of each test method, respectively.
Exceptions
This example demonstrates how exceptions thrown from a test method are caught by the intercept module and noted in the results. Any exception that is caught by the harness is assumed to indicate failure. If you want to write tests for expected exceptions, consider using our Pass/Fail_Macros#Exception_Macros exception macros
Runtime Services
These examples cover basic usage of the Runtime Test Services API (as declared in srtest.h).
Simple
This example demonstrates how to use srTestCaseSetStatus to set status, srTestCaseAddComment to add a comment, and srTEST_ADD_COMMENT_WITH_LOCATION to add a comment that automatically includes line and file information.
Dynamic
This example demonstrates how to use srTestSuiteAddSuite, srTestSuiteAddCase, srTestSuiteAddAnnotation, and srTestAnnotationAddComment for dynamic suite, test, and annotation creation in the context of a single test method.
Override
This example demonstrates how to use srTestCaseSetStatus to override the status that would otherwise be inferred from the return value.
This example demonstrates the use of printf style format strings with srTestCaseAddComment.
srTest
These examples show to how to use the stride::srTest base class for your test classes. When you publicly inherit from srTest, you get access to default testCase and testSuite members and their associated methods.
Simple
This example demonstrates the use of testCase.setStatus to set the status for test cases.
Dynamic
This example demonstrates how to use testSuite.AddSuite, testSuite.AddTest, testSuite.AddAnnotation, and testSuite.AddAnnotation.AddComment for dynamic suite, test case, and annotation creation within the context of one test method.