STRIDE Runner: Difference between revisions

From STRIDE Wiki
Jump to navigation Jump to search
No edit summary
No edit summary
Line 1: Line 1:
__FORCETOC__
[[Media:__FORCETOC__]]
== The STRIDE Test Unit Runner Application ==


The '''stride''' executable runs target-based tests from a host computer and optionally uploads results to [[STRIDE Test Space]]. The program also can be used to list the names of all Test Units that exist in a STRIDE database file.
The '''stride''' executable runs target-based tests from a host computer and optionally uploads results to [[STRIDE Test Space]]. The program also can be used to list the names of all Test Units that exist in a STRIDE database file.
Line 6: Line 5:
The stride executable is available to run on Windows XP and on Linux x86.
The stride executable is available to run on Windows XP and on Linux x86.


=== Running Tests ===
== Running Tests ==
You may specify either built-in diagnostics or user tests to be run.
You may specify either built-in diagnostics or user tests to be run.  


==== Input ====
The built-in diagnostics are useful to validate a new installation. See [[Verifying Installation]] for details.
database file


tests to run
User tests are either sample tests provided by S2 Technologies or tests that you create.


==== Output ====
=== Input ===
console output
In order to run tests, you must provide the following information to stride.exe:


local xml file
;database file
: This is the .sidb file that is created by [[s2scompile]] during the target build process. This file contains meta-information used to run tests.


test space
;device parameters
: This tells stride.exe how to connect to the target.


;tests to run
: Unless you are running the diagnostic tests (using the <tty>--diagnostics</tty> option), you must specify the Test Units that you want to run. You may also optionally specify named hierarchical suites in which to put the results of the Test Units you designate.


=== Syntax ===
By default, all Test Units in the specified database are run, and the results are all put into an unnamed root suite. It is not necessary to explicitly specify Test Units unless you want to override the default behavior.
    stride [<i>options</i>]
 
You specify the Test Units to be run by name or by wildcard.
* '''<code>*</code>''' (asterisk) matches all Test Units
* '''<code>-</code>''' (hyphen) matches all remaining Test Units (useful when putting Test Units into suites)
 
==== Test Unit Specification Rules ====
 
* Test Units are specified by name or wildcard and grouped together within parentheses, i.e. "(" and ")"
* When specifying more than one Test Unit in a group, each Test Unit name (or wildcard) must be delimited by a comma.
* The output of each Test Unit group is placed within a hierarchy of named suites. (The root suite does not have a name.)
* The suite into which a Test Unit group is placed is specified immediately before the group, e.g. ''suitepath''(''testunitgroup'')
* Hierarchical suite paths are delimited by "/" (forward slash) and are always specified starting from the root.
 
=====Examples=====
 
;<code>-r</code>
: Default behavior: run all Test Units and put results into the root-level suite.
 
;<code>-r /(*)</code>
: Same as above: run all Test Units and put results into the root-level suite.
 
;<code>-r /Suite(*)</code>
: Run all tests and put results into a suite named "Suite" that is a child of the root suite.
 
;<code>-r /(TU1,TU2,TU3)</code>
: Run the Test Units named "TU1", "TU2", and "TU3" in the designated order; put results into the root suite.
 
;<code>-r /(TU1,TU2,TU3) -r /SecondPass(TU1)</code>
: Run the Test Units named "TU1", "TU2", and "TU3" in the designated order; put results into the root suite. Then run the Test Unit named "TU1" again and put the results into a suite named "SecondPass".
 
;<code>-r /(TU1,TU2,TU3) -r /Remaining(-)</code>
: Run the Test Units named "TU1", "TU2", and "TU3" in the designated order; put results into the root suite. Then run all remaining Test Units and put the results into a suite named "Remaining".
 
;<code>-r "/Suite1/Suite2(TUa, TUb)"</code>
: Run the Test Units named "TUa" and "TUb", and put the results into a suite named "Suite2" that is a child of a suite named "Suite1" that is a child of the root. Note that we must enclose the specification in quotes since the specification contains a space.
 
;<code>-r "TUx TUy TUz"</code>
: This is a special convenience syntax. If the suite and grouping parentheses are omitted, the app runs the  the named Test Units and puts their results put into the root suite.
 
=== Output ===
Upon test completion, test output is always written as follows:
 
;console output
: A quick summary of results is written to standard output. Test counts are shown for the categories of '''passed''', '''failed''', '''in progress''', and '''not in use'''


Any warnings or errors that occur during execution are written to the standard output device.
;local xml file
: Detailed results are written to a local xml file. By default, this file is written to the directory where the input STRIDE database file is located and given the same name as the database file with an extension of ".xml". If you open this file in a web browser, an xsl transform is automatically downloaded and applied before rendering.
Prints a summary of results on the console: passed/failed/in progress/not in use


by default, xml output is written to the db directory using db name with xml extension
Optionally, you may also publish the results to your [[STRIDE Test Space]] upon test completion.


run a test
;Test Space
: Results are uploaded using your Test Space URL and login credentials. You must specify the testspace name and project name when using this option.


check program version


list database contents
== Syntax ==
    stride [<i>options</i>]


run diagnostic test
Any warnings or errors that occur during execution are written to the standard output device.


=== Options ===
== Options ==
{| border="1" cellspacing="0" cellpadding="10" style="align:left;background-color:#ffffcc;"   
{| border="1" cellspacing="0" cellpadding="10" style="align:left;background-color:#ffffcc;"   
!width="200pt"|'''Option'''
!width="200pt"|'''Option'''
Line 45: Line 90:
|-  
|-  
| '''--version'''
| '''--version'''
| Print version information.
| Prints version information to the console.
|-  
|-  
| '''--database'''  ''arg''
| '''--database'''  ''arg''
| Specifies the name of an existing STRIDE database (.sidb) file that will be used for test execution.
| Specifies the name of an existing STRIDE database (.sidb) file that will be used for test execution. This can be a relative or absolute path. If the path contains one or more spaces, it must be enclosed in quotes.
|-  
|-  
| '''--device''' ''arg''
| '''--device''' ''arg''
Line 54: Line 99:
|-
|-
| '''--timeout''' (=0)
| '''--timeout''' (=0)
| Specifies a watchdog timeout (in milliseconds) per single execution step. (0 = infinite timeout)
| Specifies a watchdog timeout (in seconds) per single test execution step. (0 = infinite timeout)
|-
|-
| '''-r''' [ --run ] arg (=/(*))
| '''-r''' [ --run ] arg (=/(*))
| Specifies a list of Test Units to execute and their order of execution with optional grouping by suite.
| Specifies a list of Test Units to execute and their order of execution with optional report grouping by suite. You may specify this option many times on the command line to include multiple groupings in a single test. A named Test Suite may be specifed more than once to run it multiple times.
* = wild card
- = remaining
For example:
  /(TU1,TU2,TU3)
  "/Suite1/Suite2(TUa, TUb)"
  /Suite(*)
  "TUx TUy TUz"


|-
|-
Line 96: Line 134:
|}
|}


===Environment Variables===
== Environment Variables ==
The following environment variables are recognized by the application and their values used as shown if set. Any value specifed by an environment variable will be overridden if the corresponding value is specified onn the command line.
The following environment variables are recognized by the application and their values used as shown if set. Any value specifed by an environment variable will be overridden if the corresponding value is specified onn the command line.


;STRIDE_DEVICE
;STRIDE_DEVICE
: specifies the --device argument
: specifies the <tt>--device</tt> argument
;STRIDE_TESTSPACE_URL  
;STRIDE_TESTSPACE_URL  
: specifies the  --testspace argument
: specifies the  --testspace argument
Line 108: Line 146:
: specifies the --space argument
: specifies the --space argument


=== Examples ===
== Examples ==
Refer to the [[Build_Tools#Compiling_multiple_files|Compiling]] section of the Build Tools Examples.
Refer to [[Running Unit Tests]] for additional examples.


=== Notes ===
== Notes ==
<references/>
<references/>
[[Category:Build_Tools]]
[[Category:Test Units]]




[[Category:Reference]]
[[Category:Reference]]

Revision as of 22:06, 27 May 2009

[[Media:]]

The stride executable runs target-based tests from a host computer and optionally uploads results to STRIDE Test Space. The program also can be used to list the names of all Test Units that exist in a STRIDE database file.

The stride executable is available to run on Windows XP and on Linux x86.

Running Tests

You may specify either built-in diagnostics or user tests to be run.

The built-in diagnostics are useful to validate a new installation. See Verifying Installation for details.

User tests are either sample tests provided by S2 Technologies or tests that you create.

Input

In order to run tests, you must provide the following information to stride.exe:

database file
This is the .sidb file that is created by s2scompile during the target build process. This file contains meta-information used to run tests.
device parameters
This tells stride.exe how to connect to the target.
tests to run
Unless you are running the diagnostic tests (using the <tty>--diagnostics</tty> option), you must specify the Test Units that you want to run. You may also optionally specify named hierarchical suites in which to put the results of the Test Units you designate.

By default, all Test Units in the specified database are run, and the results are all put into an unnamed root suite. It is not necessary to explicitly specify Test Units unless you want to override the default behavior.

You specify the Test Units to be run by name or by wildcard.

  • * (asterisk) matches all Test Units
  • - (hyphen) matches all remaining Test Units (useful when putting Test Units into suites)

Test Unit Specification Rules

  • Test Units are specified by name or wildcard and grouped together within parentheses, i.e. "(" and ")"
  • When specifying more than one Test Unit in a group, each Test Unit name (or wildcard) must be delimited by a comma.
  • The output of each Test Unit group is placed within a hierarchy of named suites. (The root suite does not have a name.)
  • The suite into which a Test Unit group is placed is specified immediately before the group, e.g. suitepath(testunitgroup)
  • Hierarchical suite paths are delimited by "/" (forward slash) and are always specified starting from the root.
Examples
-r
Default behavior: run all Test Units and put results into the root-level suite.
-r /(*)
Same as above: run all Test Units and put results into the root-level suite.
-r /Suite(*)
Run all tests and put results into a suite named "Suite" that is a child of the root suite.
-r /(TU1,TU2,TU3)
Run the Test Units named "TU1", "TU2", and "TU3" in the designated order; put results into the root suite.
-r /(TU1,TU2,TU3) -r /SecondPass(TU1)
Run the Test Units named "TU1", "TU2", and "TU3" in the designated order; put results into the root suite. Then run the Test Unit named "TU1" again and put the results into a suite named "SecondPass".
-r /(TU1,TU2,TU3) -r /Remaining(-)
Run the Test Units named "TU1", "TU2", and "TU3" in the designated order; put results into the root suite. Then run all remaining Test Units and put the results into a suite named "Remaining".
-r "/Suite1/Suite2(TUa, TUb)"
Run the Test Units named "TUa" and "TUb", and put the results into a suite named "Suite2" that is a child of a suite named "Suite1" that is a child of the root. Note that we must enclose the specification in quotes since the specification contains a space.
-r "TUx TUy TUz"
This is a special convenience syntax. If the suite and grouping parentheses are omitted, the app runs the the named Test Units and puts their results put into the root suite.

Output

Upon test completion, test output is always written as follows:

console output
A quick summary of results is written to standard output. Test counts are shown for the categories of passed, failed, in progress, and not in use
local xml file
Detailed results are written to a local xml file. By default, this file is written to the directory where the input STRIDE database file is located and given the same name as the database file with an extension of ".xml". If you open this file in a web browser, an xsl transform is automatically downloaded and applied before rendering.

Optionally, you may also publish the results to your STRIDE Test Space upon test completion.

Test Space
Results are uploaded using your Test Space URL and login credentials. You must specify the testspace name and project name when using this option.


Syntax

   stride [options]

Any warnings or errors that occur during execution are written to the standard output device.

Options

Option Description
--version Prints version information to the console.
--database arg Specifies the name of an existing STRIDE database (.sidb) file that will be used for test execution. This can be a relative or absolute path. If the path contains one or more spaces, it must be enclosed in quotes.
--device arg Specifies the parameters to be used to connect to the target device (i.e. TCP:host:port or COMport:rate:mode). For example: TCP:localhost:8000 or COM7:28800:8N1 or /dev/ttyS2:57600:8N1
--timeout (=0) Specifies a watchdog timeout (in seconds) per single test execution step. (0 = infinite timeout)
-r [ --run ] arg (=/(*)) Specifies a list of Test Units to execute and their order of execution with optional report grouping by suite. You may specify this option many times on the command line to include multiple groupings in a single test. A named Test Suite may be specifed more than once to run it multiple times.
-u [ --upload ] [=arg(=full)] Specifies an upload of results to STRIDE Test Space. Arg specifies full or incomplete/incremental/complete.
--testspace arg Specifies the root STRIDE Test Space URL and credentials. (i.e. http://host:port@user:pwd)
--project arg Specifies the name of the STRIDE Test Space project to which the results will be uploaded
--space arg Specifies the name of the STRIDE Test Space project's test space.
--name arg Specifies the name under which the results will be uploaded to STRIDE Test Space
--list Lists all Test Units in the specified STRIDE database
--diagnostics Performs a set of built-in diagnostic tests on the target. Verifying_Installation
-o [ --output ] arg Specifies the local file to which results will be written.
-O [--options_file] arg A file that contains command line options. The format is the same as the command line with the only addition that it could be split on multiple lines. A line starting with "#" symbol is ignored.

This option is necessary if the length of the command line string exceeds system limits. Otherwise it is provided only as a convenience.

Environment Variables

The following environment variables are recognized by the application and their values used as shown if set. Any value specifed by an environment variable will be overridden if the corresponding value is specified onn the command line.

STRIDE_DEVICE
specifies the --device argument
STRIDE_TESTSPACE_URL
specifies the --testspace argument
STRIDE_TESTSPACE_PROJECT
specifies the --project argument
STRIDE_TESTSPACE_SPACE
specifies the --space argument

Examples

Refer to Running Unit Tests for additional examples.

Notes