STRIDE Runner

From STRIDE Wiki
Revision as of 17:20, 2 April 2018 by Ivailop (talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

The stride executable (a.k.a. Stride Runner) runs script or native target-based tests from a desktop (Windows, Linux or FreeBSD) host computer and optionally uploads results to Testspace. 

stride [options] [TU...]

Each command line option that accepts an argument should be entered with a space between the option and its corresponding argument; i.e. 

stride -o argument --option1 argument1

Input

In order to run tests, you must provide the following information to Runner:

database file
This is the .sidb file that is created by the Build Tools during the target build process. This file contains meta-information used to run tests.
device parameters
This tells how to connect to the target.
tests to run
A set of Test Units and or Test Scripts.

Running

By default, no tests are run. You need to explicitly list them using the --run option.

Order

When explicitly specified, Tests are run in the order in which they are given on the command line.

It is possible to run all Test Units by explicitly specifying a wildcard (*). In that case they are run in alphabetical order.

Rules

  • Test Units are specified by name or wildcard
  • Scripts are specified by relative or absolute file name
  • Tests (scripts or test units) are grouped together within curly braces, i.e. "{" and "}"
  • When specifying more than one Test in a group, each Test instance must be delimited by a semicolon.
  • When a Test accepts parameters they are passed following the Test name in parentheses comma separated, i.e. "myTest(param1, param2)"
  • The output of each Test group is placed within a hierarchy of named suites. (The root suite does not have a name.)
  • The suite into which a Test group is placed is specified immediately before the group, e.g. suitepath{testunitgroup}
  • Hierarchical suite paths are delimited by "/" (forward slash) and are always specified starting from the root.
  • If the same Test is specified to be run more than once, and one or more results are to be written to the same suite, the each conflicting Test name is appended with an incrementing count in the form of "(n)". For example: Results from three runs of TestUnit "myTest" are all written to the root suite. The Tests will be reported with the names "myTest" "myTest(1)" and "myTest(2)".

Wildcard Matching

The following wildcard characters are recognized in Test Unit specifications:

  • * (asterisk) matches all Test Units
  • - (hyphen) matches all remaining Test Units (useful when putting Test Units into suites)

Examples

-r "/{*}"
Run all Test Units and put results into the root-level suite.
-r "/Suite{*}"
Run all tests and put results into a suite named "Suite" that is a child of the root suite.
-r "/{Test1;Test2;Test3}"
Run the Tests named "Test1", "Test2", and "Test3" in the designated order; put results into the root suite.
-r "/{Test1;Test2;Test3}" -r "/SecondPass{Test1}"
Run the Tests named "Test1", "Test2", and "Test3" in the designated order; put results into the root suite. Then run the Test Unit named "Test1" again and put the results into a suite named "SecondPass".
-r "/{Test1;Test2;Test3}" -r "/Remaining{-}"
Run the Test Units named "Test1", "Test2", and "Test3" in the designated order; put results into the root suite. Then run all remaining Test Units and put the results into a suite named "Remaining".
-r "/Suite1/Suite2{TestA; TestB}"
Run the Tests named "TestA" and "TestB", and put the results into a suite named "Suite2" that is a child of a suite named "Suite1" that is a child of the root. Note that we must enclose the specification in quotes since the specification contains a space.
-r "/{TestA(29); TestB(5.67, \"some text\")}"
Run the Tests named "TestA" with parameter 29 and "TestB" with parameters 5.67 and "some text", and put the results into the root suite. Note that we must enclose the specification in quotes since the specification contains spaces and embedded quotes (which needs to be escaped using backslash).
-r "TestX; TestY; TestZ"
This is a special convenience syntax. If the suite and grouping braces are omitted, the app runs the the named Test Units and puts their results put into the root suite.

Output

Upon test completion, test output is always written as follows:

console output
A quick summary of results is written to standard output. Test counts are shown for the categories of passed, failed, in progress, and not in use
local xml file
Detailed results are written to a local xml file. By default, this file is written in the current directory, named as "testresult.xml". In addition a "testresult.html" is generated that could be open in a web browser of your choice.
Testspace
Optionally, you may also publish the results to your Testspace upon test completion. Results are uploaded using your Testspace URL and login credentials. You must specify the a destination Testspace project and space names.

Options

Option Description
--database [ -d ] arg Specifies the name of an existing STRIDE database (.sidb) file that will be used for test execution.

This can be a relative or absolute path. If the path contains one or more spaces, it must be enclosed in quotes. Under Windows, both DOS and UNC names are accepted.

As a path delimiter you may use either the forward slash or backslash character. (Forward slashes are typically preferred due to the standard use of the backslash as an escape character.)

--device arg Specifies the parameters to be used to connect to the target device (i.e. TCP:host:port or COMport:rate:mode). For example:
  • TCP:localhost:8000
  • COM7:28800:8N1
  • /dev/ttyS2:57600:8N1
--timeout arg (=0) Specifies a watchdog timeout (in seconds) per single test function or test method.

Default value is 0. (0 = infinite timeout)

--run [ -r ] arg Specifies a list of Test Units or Scripts to execute and their order of execution with optional report grouping by suite. You may specify this option many times on the command line to include multiple groupings in a single test. A named Test Suite may be specified more than once to run it multiple times. See Input section above for more details.

Any nameless positional option is treated as this option.

--trace [ -t ] arg (=txt) Specifies if and how to trace on target instrumentation (functions, messages and test points/logs).

The following types of tracing formats are supported:

  • "txt" - single-entry-per-line plain text file
  • "yml" - YAML formated file
  • "-" - interactive single-entry-per-line plain text

With the exception of the interactive type all others are attached as an annotation to the output file.
Default value is "txt".

--trace_timeout arg Specifies a trace timeout (in seconds).
--upload [ -u ] arg (=start) Specifies if and how to upload the results to Testspace. Arg specifies type of upload;

The following types of uploads are supported:

  • "full" - This causes a new result set to be created and marked complete.
  • "start" - This is used to create a new result set and mark it incomplete. An upload of this type should be followed by zero or more uploads that add results and one final upload that finishes the result set.
  • "add" - This is used to add more data to an existing incomplete result set. The result set will still be marked incomplete after this upload type.
  • "finish" - This is used to add more data to an existing incomplete result set and mark it complete. This is the last step in creating a result set from multiple executions.

Default value is "start".

--testspace arg Specifies the Testspace URL to which the results will be uploaded, expressed in the following form:

user:pwd@<domain>.testspace.com/<project>/<space>
Where 'domain' is your organization's subdomain, and 'project' and 'space' are the names of related project and space.

--result arg Name of the output XML result file and optionally, when uploading, the result set name along with folder path (default "root").

The following ways to specify are supported:

  • "myfile.xml" - set both, output XML result file and result set name, to be the same.
  • "myname|" - set the result set name only, use the default "testresult.xml" as output XML result file.
  • "|myfile.xml" - set the output XML result file only, let Testspace to choose the result set name.
  • "myname|myfile.xml" - set both, result set name and output XML result file, to be different names.
  • "myname/path/to/folder|myfile.xml" - set both, result set name and output XML result file, to be different and upload under an explicit folder path.
  • "/path/to/folder|myfile.xml" - set the output XML result file, let Testspace to choose the result set name but upload under an explicit folder path.

Default value is "testresult.xml".

--list Lists all Test Units and intercepted Functions in the specified Stride database to standard out.
--diagnostics Performs a set of built-in diagnostic tests on the target. See Running Diagnostics for details.
--log_level arg (=warn) Controls the target source Test Logs level.

The following levels are supported:

  • "all" - enables all logging
  • "info" - enables only INFO and higher
  • "warn" - enables only WARN and higher
  • "err" - enables only ERROR and higher
  • "off" - disables all logging

Default value is "warn".

--options_file arg Specifies a file from which the program reads command line options. The format is the same as the command line except that options may be split across multiple lines. Lines in an options file that begin with the character '#' are ignored.
--version Prints version information to the console.
--dry Dry run, just validate the input without device connection or test execution.


Using a Proxy

If you access the Internet via an HTTP proxy, you must set the environment variables HTTP_PROXY and HTTPS_PROXY to specify the name and port of the proxy server.

Symptoms of needing to specify a proxy is the following errors: 

Unable to proceed. Invalid testspace address or credentials. Couldn't access the specified company. 
Failed performing request: [7] Couldn't connect to server

Unable to proceed. Failed performing request: [6] Couldn’t resolve host name

Define the environment variables in format host:port where host:port corresponding to your proxy server. For example, on Windows: 

> set HTTP_PROXY=myproxy:8765
> set HTTPS_PROXY=myproxy:8765

would instruct the stride runner to use the proxy named myproxy on port 8765 to communicate via HTTP and HTTPS protocols.

If your proxy requires authentication use the format username:password@host:port where username and password are your credentials.

Retrieved from "https://www.stridewiki.com/index.php?title=STRIDE_Runner&oldid=14691"