Running Tests (old)

STRIDE Framework tests (script and native target implementations) are controlled by a host computer, physically connected to the target via a configurable communication channel (TCP/IP or serial port). This article describes some common usage patterns and presents several examples.

Prerequisites

Host

Tests are initiated on the host by running the program stride on the host computer (installed as part of STRIDE Desktop Installation). See the linked article for reference information.

In order for stride to run your tests, the STRIDE database (.sidb) file that was produced as part of the target build must be accessible to the program via the host filesystem. This is trivial to accomplish when the target is built on the host computer (as when building and running tests off-target) or when the host and target computers are running the same operating system and are connected via LAN. But for dissimilar operating systems, you will need to mount the target filesystem on the host computer. (Many of our customers use Samba to accomplish this.)

Target

• Target transport configured
• Target built with STRIDE library and some STRIDE source instrumentation (detials can be found here). If any native test units have been written, these must be included in the build also.
• Target app running

Running Tests

The following command line illustrates how to run some STRIDE tests without publishing results to STRIDE Test Space.

We assume the following:

• We are testing against a target build that contains the following native code test units: myTest1, myTest2 and myTest3.
• I also want to run host based tests implemented in a perl test module, myScriptTests.pm.
• The target device is running, and is available on the LAN via TCP/IP at 192.168.0.53

The command line to run the desired tests would be:

stride --run "myTest1;myTest2;myTest3" --run myScriptTests.pm --database "/my/path/targetApp.sidb" --device TCP:192.168.0.53:8000

A couple of things to note:

• If you are running tests of a shared location and others may be running tests using the same referenced database, you should specify a unique output file location (via the -o option) to prevent overwriting the output file.
• If you will be running many tests against a specific database and/or device, consider setting environment variables to the corresponding values, or put command line options into an options file (via the -O option).
• You can refer to the device address using a DNS name if desired (e.g. TCP:testcomputer:8000)

Running All Tests

By default, stride will not run any tests if --run options are not specified. A simple way to run all target based test units (tests implemented in native code on the device) is to specify the wildcard parameter to stride - specifically:

stride --database TestApp.sidb --device TCP:128.0.0.1:8000 --run "*"

This will cause all test units that have been captured in the STRIDE database to be executed.

There is no way to run all perl test modules since we don't capture information about these in the STRIDE database. However, if you organize your test modules in a directory hierarchy, the runner provides a convenient syntax for executing all test modules in a directory - for example:

stride --database TestApp.sidb --device TCP:128.0.0.1:8000 --run "path/to/my_directory"

will recursively search my_directory for script test modules (identified by file extension) and execute each as it is found.

Running A Subset of Tests

The stride test runner allows you to specify a subset of the test units in an existing target build.

For example, assume that you have an .sidb file produced from your target build that has many test units defined. You want to run only three of the test units to reproduce a bug, say. We'll assume that the test units you want to run are named MyTest1, MyTest2, and MyTest3. Further, we'll assume that the environment variable STRIDE_DEVICE is set to reflect your device parameters so that the --device argument can be omitted.

The following command line will run the three tests in the order shown, putting their results into the root test suite.

stride --database TestApp.sidb --run /{MyTest1;MyTest2;MyTest3}

Or, if your test results are to be published in the root suite, you can use this simplified syntax:

stride --database TestApp.sidb MyTest1 MyTest2 MyTest3

You can automate the running of a subset of test units who's names match a text pattern using the stride runner's Listing Test Units capability and a small perl script as shown below.

while( <> ) {
    if( /myPrefix_*/) {
        print "-r $_";
    }
}

The perl script reads from stdin and attempts to match the line to the regular expression myPrefix_. If a match is detected, the line is written to stdout prefixed with -r ,

If you save the perl script to a file named myTests.pl, you can automate the running of tests that match your prefix as follows:

stride --list --database TestApp.sidb | perl myTests.pl > tests.txt
stride --database TestApp.sidb --options_file tests.txt

Publishing Test Results

Saving Results to the Local Disk

When stride is executed to run tests, the following files are written to the current directory. By default, they are named after the specified database. You can specify a different name and location by using the -o option when you run stride. If these files already exist, they are overwritten.

[database_name].xml

Raw xml test output.

[database_name].xsl

Stylesheet to format the xml file for viewing in a web browser. (This file is automatically downloaded only if you are connected to the Internet.)

Displaying Local Results Data

If you open the xml file in a browser, the xsl stylesheet (if present) will be automatically applied to the xml so that the data is rendered in html within the browser.

If you are using Google Chrome please see Browser Compatibility.

Publishing Results to STRIDE Test Space

stride can also be instructed to upload test results to STRIDE Test Space.

Example: Desktop Testing with Publication to Test Space

This example demonstrates the running of specified target tests followed by publication of the results to STRIDE Test Space.

Here, we create an options file since the command line is getting very long. (We could put several of these option values in environment variables instead if desired.)

The option file is named myOptions.txt, is located in the current directory and contains the following text:

--database "/my/path/targetApp.sidb"
--device TCP:192.168.0.53:8000
--run "/{myTest1;myTest2; myTest3}"
--upload
--testspace username:password@mycompany.stridetestspace.com
--project "Super Repeater"
--space "Input Processor"
--output myoutputfile.xml

The command line is then:

stride --options_file "myOptions.txt"

Note: In order for publishing to succeed, the username/password must be correct and the project and space must already exist. If you are using TestSpace for the first time, you should check with your administrator about how to create projects and spaces.

Publishing Pre-Existing Test Report Files

Stride.exe can be used to upload to Test Space one or more pre-existing report files only (without running tests).

To accomplish this specify the publishing-related parameters but do not specify the --device parameter or any --run parameters when you run stride. The file to be uploaded is specified by the --output parameter.

If you want aggregate more than one .xml file to a single Test Space result set, run stride once for each file to be uploaded, specifying the --upload parameter accordingly.

Note that stride will use any environment variables that are in effect, so if you have a non-empty value set for STRIDE_DEVICE it must be set to empty in your terminal session before upload-only will be performed. Similarly, environment variables STRIDE_TESTSPACE_URL, STRIDE_TESTSPACE_PROJECT, and STRIDE_TESTSPACE_SPACE will be used if non-empty. (These can be overridden on the command line.)

Examples

Upload a single report file

stride --testspace user:pwd@s2technologies.stridetestspace.com --project Sandbox --space "Tim Space" --name RSName --upload --output test.xml

Upload multiple report files to a single result set

Run stride once for each file to be aggregated. Note argument to --upload parameter.

stride --testspace user:pwd@s2technologies.stridetestspace.com --project Sandbox --space "Tim Space" --name RSName --upload start --output test1.xml

stride --testspace user:pwd@s2technologies.stridetestspace.com --project Sandbox --space "Tim Space" --name RSName --upload add --output test2.xml

stride --testspace user:pwd@s2technologies.stridetestspace.com --project Sandbox --space "Tim Space" --name RSName --upload add --output test3.xml

stride --testspace user:pwd@s2technologies.stridetestspace.com --project Sandbox --space "Tim Space" --name RSName --upload finish --output test4.xml

Other Resources

A technique to organize tests into suites is shown here.

For full details on stride command line parameters, please see STRIDE Runner.