Studio:Test Runners

STRIDE provides three command line utilities for executing tests.

TestUnitRun.pl

TestUnitRun is a light-weight utility for command line execution of STRIDE Test Units.

Usage

 TestUnitRun.pl [-d foo.sidb -r MyTestUnit -t 10 -v -f Directory -o OrderFile]

Parameters

-d | --database = database.sidb

Sets the STRIDE database to be used for execution. The database file is produced by the STRIDE build tools. If no database is specified, the current directory is searched and the first .sidb file that is found (in filesystem order) is used.

-o | --order-file = order.txt

Specifies an input file to set the items and order of execution. This is a simple, human readable text file containing Test Unit names separated by any combination of newlines, semicolons, and commas. Each item specified in this file will be fetched from the STRIDE type database - a warning will be issued if any are not found. Special suite path tokens are supported - any item surrounded by curly braces {} will be interpreted as a suite path specification. All test unit items that follow a suite specification will be placed in the suite indicated. The suite can use the forward slash / to specify a hierarchy of suites. An empty suite specification {} causes subsequent test unit items to be placed in the root test suite for the run. This option can be specified multiple times in which case each order file will be processed as described.

-f | --filesystem-order= Directory

Specifies an filesystem directory that contains one or more source files that were contain the test unit SCL specifications. The runnner will attempt to infer execution order for the corresponding test units based on the directory hierarchy of the files that contain the SCL specifications for the test units. This argument can be specified multiple times -- each provided directory will be examined for source files that correspond to test unit objects in your STRIDE database. This option cannot be specified when an order-file is specified. If one or more filesystem-order directories are specified, then ONLY test units that are associated with source files in those directories will be executed. Any test units that are captured in other source files outside of these file trees will be ignored.

-r | --run = TestUnitName

Specifies a specific test unit to execute. The database file is searched for a Test Unit with this name - if none is found, a warning is issued. This option may be specified multiple times to run multiple Units. If an order file has been specified then this argument is ignored.

-t | --timeout = TimeoutVal

Specifies a timeout (in seconds) to use when attempting to connect to the device. If it is unable to connect in TimeoutVal seconds, the script will issue an error and exit. The default value is 5 seconds.

-v | --view-report

Flag indicating whether or not the default registered HTML handler should be invoked for the final report file. Default behavior is that the report is not opened at the end of execution.

Prerequisites

The Config::IniFiles package must be installed for this script to run. If using an ActiveState perl distribution, the command 'ppm install config-inifiles' will insure that it is installed.

The Target connection parameters are specified in a [STRIDE_DIR]\bin\transport.cfg configuration file. The default values need to be adjusted to the test environment.

WorkspaceRun.pl

WorkspaceRun is a light-weight utility for executing the scripts and/or folders in a workspace. This utility automates the execution of scripts and/or folders in one or more workspaces. If no workspace file is specified, it will search the current directory tree and run all ssw files that are found. If no top-level folders are specified for exclusion, the entire workspace will be executed. This utility does no special process or timeout monitoring.

The results of the execution are written to the report file (html) specified in the workspace settings. A simple totals summary of the results is also printed to stdout upon completion of each workspace.

Usage

 WorkspaceRun.pl [--workspace=tests.ssw --exclude=myFolder]

Parameters

-w | --workspace = WorkspaceName.ssw

Sets the workspace to be opened. If no workspace is specified, the current directory is searched and all ssw files will be opened, in file-system order. Multiple workspace switches can be used to specify more than one workspace to open.

-x | --exclude= path/my/folder

Sets a logical folder in studio to be excluded from execution. Multiple folder switches can be used to specify more than one folder to skip.

TestRunner.pl

This utility automates the execution of multiple workspaces using Stride Studio's automation and testing framework. It searches in the current directory tree for all runnable files (STRIDE workspaces and .s2tc files - see below) and attempts to execute the found items.

This utility is designed for robust execution and can continue even when errors occur in the execution. Workspace timeouts are monitored and the entire STRIDE infrastructure is restarted if workspaces fail to complete execution in the specified amount of time.

TestRunner maintains the test results and provides options to control uploading the results to a STRIDE Portal. There is also preliminary support for executing stand-alone scripts - currently only perl, jscript, vbscript and batch files are supported (.pl, .js, .vbs, and .bat respectively).

Usage

 TestRunner.pl [OPTIONS]

Parameters

--reportfile="reportfilename.html"

Overrides the default report file name (TestResults.html). If youspecify a full path, it will be honored, otherwise the file will be created in a time-stamped subdirectory of the working (starting) directory.

--datafile="datafilename.xml"

Overrides the default data file name (TestData.xml). If you specify a full path, it will be honored, otherwise the file will be created in a time-stamped subdirectory of the working (starting) directory).

--directory="some\dir"

Specifies which directory to start running workspaces in. Defaults to the current directory.

--file="some\file.ssw"

Specifies a specific file to execute. If this option is specified, only the specified file will be executed. This option only supports runnable file types: stride workspaces and s2tc files, currently.

--resultdir="some\dir"

Specifies an alternate directory for the test results (data and reports). Defaults to a time-stamped directory under the current directory.

--database="C

\\workspaces\\master\\master.sidb"

Allows user to specify a single database to use for all workspaces. The full path to a database file must be given. If this option is not used, the default database is used for each workspace.

--filter=path\to\filter.pl

Specifies a perl script file that will be used at runtime to further filter the file-set that is executed. If this script is specified and exists, for each candidate workspace file that the script finds, it will attempt to call the UseFile function in that script (the script is loaded using 'require' runtime loading). If the UseFile function exists and returns 0, the file will be skipped - otherwise the workspace will be executed.

--timeout=n

Allows user to specify a default workspace timeout value to use (in seconds). This value is only used for any workspace that does not already have a specified timeout value. Specify a value of 0 if you want this script to wait forever on workspaces.

--cleanup-delay=n

Allows user to specify the time to wait after each workspace exucution before checking for processes to cleanup. If this is not specified, a default delay of 2 seconds will be used.

--nocontinue

Indicates that the script should cease execution upon a timeout error. The default behavior is to report timeout errors and continue.

--watchlog

Indicates that the script should check the StrideErrLog.txt file for any error messages after each workspace and add any error messages found to the report.

--full-restart

Causes all relevant S2 processes to be shut-down in between each workspaces that is executed. The default behavior is that we leave the S2 Reporter and Host processes running between tests. This option assures the cleanest possible process environment at the start of each test at the expense of some time lost in between each test as the reporter results are saved and reloaded.

--screenshot

Causes the script to generate screen shot images whenever the workspace exceeds the allowable run time (timeout value). This will allow you to see any messages being displayed by S2 executables prior to being killed. Please note the special installation requirements in Prerequisites section.

--portal-location="servername"

Specifies a resolvable address of a valid STRIDE Portal instance (a webserver running the STRIDE Portal software). Defaults to 'localhost'.

--portal-path="/some/path"

Specifies the portal path to use when uploading results to the STRIDE Portal.

--portal-space="myspace"

Specifies the portal space name to use when posting results to the STRIDE Portal. Defaults to public.

--portal-user="username"

Specifies the portal user name to use when posting results to the STRIDE Portal. Defaults to public.

--portal-pass="password"

Specifies the portal password to use when posting results to the STRIDE Portal. Defaults to public.

--noportal

Causes the script to skip the uploading of test results. The default behavior is to attempt to upload results whenever they are created.

Prerequisites

If you intend to use the --screenshot option (see above), you will also need to install the ImageMagick libraries and the Win32::Screenshot module. A binary distribution of the ImageMagick libraries is available at http://www.imagemagick.org/script/binary-releases.php. When installing the ImageMagick distribution, be sure to select the option to install the perl modules and the development headers (we recommend just installing everything in the distribution). The Win32::Screenshot package can be obtained from CPAN (www.cpan.org). Win32::Screenshot does require one source file to be compiled, so you must have a working development environment on your machine (Visual Studio or the VisualStudio .NET Toolkit, e.g.) when you make/install it.

Standalone Script Execution

TestRunner contains provisional support for standalone script execution. In addition to looking for all STRIDE Studio workspaces to execute (files with .ssw extension), we also look for files with an .s2tc extension (S2 test configuration files).

The s2tc files must contain xml of the following form:

<?xml version="1.0" encoding="UTF-8"?>
<StrideTestConfiguration version="1.0">
  <file path="test.js" command="exec cmd" command_arguments="args" script_arguments="args" timeout="20"/>
  ...
</StrideTestConfiguration>

File Node Attributes

path="script.pl"

This is the only attribute that is required as it indicates what script file to attempt to run. This file can be specified as a full system path (absolute path) or a relative path (relative to the location of the .s2tc file).

command="C

\path\to\script\engine.exe"

Optional command to use to execute the given script file. If no command is explicitly specified, we will attempt to infer the interpreter to use from the extension of the script file given in the path attribute. Currently we only can infer the interpreter for perl, jscript, vbscript and batch files.

command_arguments="args"

Optional additional arguments to specify to the command (script engine). These arguments will be passed to the script engine before the script file.

script_arguments="args"

Optional additional arguments to pass to the script - these arguments are appended to the execution command following the name of the script.

timeout="10"

Optional timeout value (in seconds). Each script will be allowed to run for the specified timeout (or a default of 10 minutes) and will be killed if it runs for longer than the timeout.

NOTE: the current test path (for reporting) is made available to scripts via the S2TC_TESTPATH environment variable.