Studio:Test Runners

TestRun.pl

TestRun is a light-weight utility for executing the scripts and/or folders in a workspace. This utility automates the execution of scripts and/or folders in one or more workspaces. If no workspace file is specified, it will search the current directory tree and run all ssw files that are found. If no top-level folders are specified for exclusion, the entire workspace will be executed.

Usage

 TestRun.pl [--workspace=tests.ssw --exclude=myFolder]

Parameters

-w | --workspace = WorkspaceName.ssw

Sets the workspace to be opened. If no workspace is specified, the current directory is searched and all ssw files will be opened, in file-system order. Multiple workspace switches can be used to specify more than one workspace to open.

-f | --folder = path/my/folder

Sets a logical folder in studio to be executed. Multiple folder switches can be used to specify more than one folder to run.

-s | --script = myscript.pl

Sets a script file in studio to be executed. Multiple script switches can be used to specify more than one script to run.

STRIDE_TestRunner.pl

This utility automates the execution of multiple workspaces using Stride Studio's automation and testing framework. It searches in the current directory tree for all runnable files (STRIDE workspaces and .s2tc files - see below) and attempts to execute the found items. This utility is designed for robust execution and can continue even when errors occur in the execution. Workspace timeouts are monitored and the entire STRIDE infrastructure is restarted if workspaces fail to complete execution in the specified amount of time. STRIDE_TestRunner maintains the test results and provides options to control uploading the results to a STRIDE Portal. There is also preliminary support for executing stand-alone scripts - currently only perl, jscript, vbscript and batch files are supported (.pl, .js, .vbs, and .bat respectively).

Usage

 STRIDE_TestRunner.pl [OPTIONS]

OPTIONS

   --reportfile="reportfilename.html"
       Overrides the default report file name (TestResults.html). If you
       specify a full path, it will be honored, otherwise the file will be
       created in a time-stamped subdirectory of the working (starting)
       directory.

   --datafile="datafilename.xml"
       Overrides the default data file name (TestData.xml). If you specify
       a full path, it will be honored, otherwise the file will be created
       in a time-stamped subdirectory of the working (starting) directory).

   --directory="some\dir"
       Specifies which directory to start running workspaces in. Defaults
       to the current directory.

   --file="some\file.ssw"
       Specifies a specific file to execute. If this option is specified,
       only the specified file will be executed. This option only supports
       runnable file types: stride workspaces and s2tc files, currently.

   --resultdir="some\dir"
       Specifies an alternate directory for the test results (data and
       reports). Defaults to a time-stamped directory under the current
       directory.

   --database="C:\\workspaces\\master\\master.sidb"
       Allows user to specify a single database to use for all workspaces.
       The full path to a database file must be given. If this option is
       not used, the default database is used for each workspace.

   --filter=path\to\filter.pl
       Specifies a perl script file that will be used at runtime to further
       filter the file-set that is executed. If this script is specified
       and exists, for each candidate workspace file that the script finds,
       it will attempt to call the UseFile function in that script (the
       script is loaded using 'require' runtime loading). If the UseFile
       function exists and returns 0, the file will be skipped - otherwise
       the workspace will be executed.

   --timeout=n
       Allows user to specify a default workspace timeout value to use (in
       seconds). This value is only used for any workspace that does not
       already have a specified timeout value. Specify a value of 0 if you
       want this script to wait forever on workspaces.

   --cleanup-delay=n
       Allows user to specify the time to wait after each workspace
       exucution before checking for processes to cleanup. If this is not
       specified, a default delay of 2 seconds will be used.

   --nocontinue
       Indicates that the script should cease execution upon a timeout
       error. The default behavior is to report timeout errors and
       continue.

   --watchlog
       Indicates that the script should check the StrideErrLog.txt file for
       any error messages after each workspace and add any error messages
       found to the report.

   --full-restart
       Causes all relevant S2 processes to be shut-down in between each
       workspaces that is executed. The default behavior is that we leave
       the S2 Reporter and Host processes running between tests. This
       option assures the cleanest possible process environment at the
       start of each test at the expense of some time lost in between each
       test as the reporter results are saved and reloaded.

   --screenshot
       Causes the script to generate screen shot images whenever the
       workspace exceeds the allowable run time (timeout value). This will
       allow you to see any messages being displayed by S2 executables
       prior to being killed. Please note the special installation
       requirements in PREREQUISITES above.

   --portal-location="servername"
       Specifies a resolvable address of a valid STRIDE Portal instance (a
       webserver running the STRIDE Portal software). Defaults to
       'localhost'.

   --portal-path="/some/path"
       Specifies the portal path to use when uploading results to the
       STRIDE Portal.

   --portal-space="myspace"
       Specifies the portal space name to use when posting results to the
       STRIDE Portal. Defaults to public.

   --portal-user="username"
       Specifies the portal user name to use when posting results to the
       STRIDE Portal. Defaults to public.

   --portal-pass="password"
       Specifies the portal password to use when posting results to the
       STRIDE Portal. Defaults to public.

   --noportal
       Causes the script to skip the uploading of test results. The default
       behavior is to attempt to upload results whenever they are created.

Prerequisites

If you intend to use the --screenshot option (see above), you will also need to install the ImageMagick libraries and the Win32::Screenshot module. A binary distribution of the ImageMagick libraries is available at http://www.imagemagick.org/script/binary-releases.php. When installing the ImageMagick distribution, be sure to select the option to install the perl modules and the development headers (we recommend just installing everything in the distribution). The Win32::Screenshot package can be obtained from CPAN (www.cpan.org). Win32::Screenshot does require one source file to be compiled, so you must have a working development environment on your machine (Visual Studio or the VisualStudio .NET Toolkit, e.g.) when you make/install it.

Standalone Script Execution

STRIDE_TestRunner contains provisional support for standalone script execution. In addition to looking for all STRIDE Studio workspaces to execute (files with .ssw extension), we also look for files with an .s2tc extension (S2 test configuration files).

The s2tc files must contain xml of the following form:

 <?xml version="1.0" encoding="UTF-8"?>
 <StrideTestConfiguration version="1.0">
   <file path="test.js" command="exec cmd" command_arguments="args" script_arguments="args" timeout="20"/>
   ...
 </StrideTestConfiguration>

File Node Attributes

path="script.pl"

This is the only attribute that is required as it indicates what script file to attempt to run. This file can be specified as a full system path (absolute path) or a relative path (relative to the location of the .s2tc file).

command="C

\path\to\script\engine.exe"

Optional command to use to execute the given script file. If no command is explicitly specified, we will attempt to infer the interpreter to use from the extension of the script file given in the path attribute. Currently we only can infer the interpreter for perl, jscript, vbscript and batch files.

command_arguments="args"

Optional additional arguments to specify to the command (script engine). These arguments will be passed to the script engine before the script file.

script_arguments="args"

Optional additional arguments to pass to the script. These arguments will be passed to the script file before the final argument which is the path of the current s2tc file in the test hierarchy. This test hierarchy path is automatically passed to every script by the framework.

timeout="10"

Optional timeout value (in seconds). Each script will be allowed to run for the specified timeout (or a default of 10 minutes) and will be killed if it runs for longer than the timeout.