Studio:Reporting Sample

Introduction

The following content relates to the sample files and workspaces installed in %STRIDE_DIR%\Samples\Scripting\Reporter. This sample consists of a Visual Studio workspace for building an off-target simulator, sample source code and test scripts, and a STRIDE workspace for sample code execution.

Getting Started

To begin, open the Visual Studio Solution file in the sample directory. This solution (and corresponding project) were created for Visual Studio 2005. If you have a later version of Visual Studio installed, you should be able to open this solution and it will be automatically upgraded if necessary. If you do not currently have any version of Visual Studio, we recommend that you install the current free version of Visual Studio Express.

Once you have successfully opened the solution, rebuild it. The build process has custom STRIDE build rules integrated and will produce a STRIDE database, intercept module source files, and an off-target simulator application that incorporates the sample source code.

Once the build is complete, perform the following steps to run the sample scripts in the workspace:

1. launch the off-target simulator, Reporter_Sample.exe. This will run in a standard console window.
2. open a command prompt window and change to this sample's directory.
3. at the command prompt, run the command WorkspaceRun.pl -x setup -x teardown. This will execute all of the sample test scripts in the workspace and displays the results. You may open a browser and display the report written to the Reporter_Sample.html file in the sample's directory .
4. quit the Reporter_Sample.exe application by typing 'q' in its console window.

Overview

The STRIDE Reporter object provides a means to organize and collect test results data, then publish results in a format of your choosing. The STRIDE Studio application integrates a reporter object with its collection of test scripts, and this sample's Studio workspace defines the hierarchy of the test scripts that will be the resulting hierarchy of the test suites and test cases in the generated report. The following describes the contents of the test report in more detail.

Report Organization

As stated earlier, the Reporter stores test results hierarchically. At the top level, it holds a collection of one or more child Suites. These Suites--in turn--hold a collection of zero or more child Tests, a second collection of zero or more Suites, and a third collection of zero or more Annotations.

Each Test has a collection of zero or more Comments, and each Annotation has a collection of zero or more Comments.

The diagram at the right shows an example hierarchy.

Reporter Object Hierarchy

The Suite Object

The Suite object is used to group related Tests (and Annotations--if present) so that their results can be summarized on an output report.

The Suite object exposes a set of properties, the most important are listed below.

• Name
• Description

The Test Object

The Test object represents a specific test that is run. The result of a test is reflected in its Status property ("PASS"/"FAIL").

The most important properties of the Test object are listed below.

• Name
• Description
• Status

The Comment Object

The Comment object allows you to attach a text comment to a Test or Annotation. Comments are useful for logging test script flow and other test information.

The Comment object exposes two properties listed below.

• Comment
• Label

The Annotation Object

The Annotation object is used when you want to associate a file or a URI with a Suite. (e.g. a screen shot or other data file).

The Annotation object is designed for use with STRIDE Portal, and will not be covered further here. For more information, see the Annotation item section of the Reporter reference article.

Sample Scripts

The scripts that demonstrate the use of the reporter object are in the workspace test folder, in the Files tab. Examples scripts in JScript (NegativeTests.js) and Perl (PositiveTests.pl) are provided. Each uses the sample test function IsLeapYear, passing in integral values and using the return value to create the content of the test suite. The scripts use the AutoScript objects Functions collection to call the test function (see the AutoScript_Samples article to learn more about using the AutoScript object with scripts.)

The following describes the highlights of how the Reporter object is used in the sample scripts.

Creating Suites

Suites can be created from a script, but it's unnecessary when running within STRIDE Studio, as a Suite is created for you automatically for each folder under Script Files in the workspace tree.

For convenience, the automatically-created Suite object that is a script's immediate parent is injected into the script's namespace. This Suite is accessed in the sample scripts by using the pre-defined testSuite object.

Creating Tests

The Test objects are created and added to the parent Suite from within each script. A typical pattern is to implement one test per script, but you can create as many Tests as you like in a single script.

perl

# create a new test in the parent suite
my $test = $main::testSuite->Tests->Add("Leap Year Positive Tests");
$test->{Description} = "Tests the IsLeapYear functionality";

JScript

// create a new test in the parent suite
var test = testSuite.Tests.Add("Leap Year Negative Tests");
test.Description = "Tests the IsLeapYear functionality";

Creating Comments

perl

# add a new comment to the test report with the passed-in label
my $comment = $test->Comments->Add("Testing $input[$i]");
 
...
 
# test the return value; note that the value is returned as the name of the enumerator
if ($f->ReturnValue eq "YES") {
    # set the comment text
    $comment->{Comment} = "Pass";
}
else {
    # set the comment text
    $comment->{Comment} = "Fail";
    # indicate that at least one test failed
    $testResult = 1;
} 

JScript

// add a new comment to the test report with the passed-in label
var comment = test.Comments.Add("Testing " + arInput[nIndex]);

...

// test the return value; note that the value is returned as the name of the enumerator
if (f.ReturnValue == "NO") {
    comment.Comment = "Pass";
}
else {
    comment.Comment = "Fail";
    testResult = false;
}