Studio:Test Runners

STRIDE provides three command line utilities for executing tests.

WorkspaceRun.pl

WorkspaceRun is a light-weight utility for executing the scripts and/or folders in a STRIDE Studio workspace. For more information about it read this.

TestRunner.pl

This utility automates the execution of multiple workspaces using STRIDE Studio's automation and testing framework. It searches in the current directory tree for all runnable files (STRIDE workspaces and .s2tc files - see below) and attempts to execute the found items.

This utility is designed for robust execution and can continue even when errors occur in the execution. Workspace timeouts are monitored and the entire STRIDE infrastructure is restarted if workspaces fail to complete execution in the specified amount of time.

TestRunner maintains the test results and provides options to control uploading the results to a STRIDE Portal. There is also preliminary support for executing stand-alone scripts - currently only perl, jscript, vbscript and batch files are supported (.pl, .js, .vbs, and .bat respectively).

Usage

 TestRunner.pl [OPTIONS]

Parameters

--reportfile="reportfilename.html"

Overrides the default report file name (TestResults.html). If youspecify a full path, it will be honored, otherwise the file will be created in a time-stamped subdirectory of the working (starting) directory.

--datafile="datafilename.xml"

Overrides the default data file name (TestData.xml). If you specify a full path, it will be honored, otherwise the file will be created in a time-stamped subdirectory of the working (starting) directory).

--directory="some\dir"

Specifies which directory to start running workspaces in. Defaults to the current directory.

--file="some\file.ssw"

Specifies a specific file to execute. If this option is specified, only the specified file will be executed. This option only supports runnable file types: stride workspaces and s2tc files, currently.

--resultdir="some\dir"

Specifies an alternate directory for the test results (data and reports). Defaults to a time-stamped directory under the current directory.

--database="C

\\workspaces\\master\\master.sidb"

Allows user to specify a single database to use for all workspaces. The full path to a database file must be given. If this option is not used, the default database is used for each workspace.

--filter=path\to\filter.pl

Specifies a perl script file that will be used at runtime to further filter the file-set that is executed. If this script is specified and exists, for each candidate workspace file that the script finds, it will attempt to call the UseFile function in that script (the script is loaded using 'require' runtime loading). If the UseFile function exists and returns 0, the file will be skipped - otherwise the workspace will be executed.

--timeout=n

Allows user to specify a default workspace timeout value to use (in seconds). This value is only used for any workspace that does not already have a specified timeout value. Specify a value of 0 if you want this script to wait forever on workspaces.

--cleanup-delay=n

Allows user to specify the time to wait after each workspace exucution before checking for processes to cleanup. If this is not specified, a default delay of 2 seconds will be used.

--nocontinue

Indicates that the script should cease execution upon a timeout error. The default behavior is to report timeout errors and continue.

--watchlog

Indicates that the script should check the StrideErrLog.txt file for any error messages after each workspace and add any error messages found to the report.

--full-restart

Causes all relevant S2 processes to be shut-down in between each workspaces that is executed. The default behavior is that we leave the S2 Reporter and Host processes running between tests. This option assures the cleanest possible process environment at the start of each test at the expense of some time lost in between each test as the reporter results are saved and reloaded.

--screenshot

Causes the script to generate screen shot images whenever the workspace exceeds the allowable run time (timeout value). This will allow you to see any messages being displayed by S2 executables prior to being killed. Please note the special installation requirements in Prerequisites section.

--portal-location="servername"

Specifies a resolvable address of a valid STRIDE Portal instance (a webserver running the STRIDE Portal software). Defaults to 'localhost'.

--portal-path="/some/path"

Specifies the portal path to use when uploading results to the STRIDE Portal.

--portal-space="myspace"

Specifies the portal space name to use when posting results to the STRIDE Portal. Defaults to public.

--portal-user="username"

Specifies the portal user name to use when posting results to the STRIDE Portal. Defaults to public.

--portal-pass="password"

Specifies the portal password to use when posting results to the STRIDE Portal. Defaults to public.

--noportal

Causes the script to skip the uploading of test results. The default behavior is to attempt to upload results whenever they are created.

Prerequisites

This component is written in the Perl scripting language and therefore requires that Perl be installed. See Perl requirements for more information.

If you intend to use the --screenshot option (see above), you will also need to install the ImageMagick libraries and the Win32::Screenshot module. A binary distribution of the ImageMagick libraries is available at http://www.imagemagick.org/script/binary-releases.php. When installing the ImageMagick distribution, be sure to select the option to install the perl modules and the development headers (we recommend just installing everything in the distribution). The Win32::Screenshot package can be obtained from CPAN (www.cpan.org). Win32::Screenshot does require one source file to be compiled, so you must have a working development environment on your machine (Visual Studio or the VisualStudio .NET Toolkit, e.g.) when you make/install it.

Standalone Script Execution

TestRunner contains provisional support for standalone script execution. In addition to looking for all STRIDE Studio workspaces to execute (files with .ssw extension), we also look for files with an .s2tc extension (S2 test configuration files).

The s2tc files must contain xml of the following form:

<?xml version="1.0" encoding="UTF-8"?>
<StrideTestConfiguration version="1.0">
  <file path="test.js" command="exec cmd" command_arguments="args" script_arguments="args" timeout="20"/>
  ...
</StrideTestConfiguration>

File Node Attributes

path="script.pl"

This is the only attribute that is required as it indicates what script file to attempt to run. This file can be specified as a full system path (absolute path) or a relative path (relative to the location of the .s2tc file).

command="C

\path\to\script\engine.exe"

Optional command to use to execute the given script file. If no command is explicitly specified, we will attempt to infer the interpreter to use from the extension of the script file given in the path attribute. Currently we only can infer the interpreter for perl, jscript, vbscript and batch files.

command_arguments="args"

Optional additional arguments to specify to the command (script engine). These arguments will be passed to the script engine before the script file.

script_arguments="args"

Optional additional arguments to pass to the script - these arguments are appended to the execution command following the name of the script.

timeout="10"

Optional timeout value (in seconds). Each script will be allowed to run for the specified timeout (or a default of 10 minutes) and will be killed if it runs for longer than the timeout.

NOTE: the current test path (for reporting) is made available to scripts via the S2TC_TESTPATH environment variable.