Creating And Using Baselines

Introduction

Baselines represent expectations of test results. Baseline data consists of suites and test cases that are used for comparing to other result sets to determine the quality of the results. In a simple test-bed that has only passing tests, baselines are not typically needed since any test failure can be considered a regression - thus regression is easy to determine by simple inspection. As test coverage becomes more extensive, however, it's inevitable that some test failures will be deemed acceptable, at least for a defined period of time (say for an upcoming release cycle). In these cases, it's necessary to define a baseline that includes the desired minimum results, including the acceptable failures. This baseline data can then be compared with each new result set to quickly determine if any result sets are different than expected.

Types

STRIDE Testspace supports two basic types of baseline comparison - fixed and sequential.

Sequential

Sequential baseline comparison simply compares each result set to the previous result set (previous based on upload order). This kind of comparison can be useful for stable test-beds that are expected to produce the same results with each run of the tests. If your test-bed is changing regularly, has inherent instability (variation in results), or is under active development, then this kind of comparison offers little value. Sequential comparison can be activated for any test space by editing the testspace settings so as to enable the auto compare sequentially property.

This screencast demonstrates this concept:

<dhflashplayer>file=TS_BaselineSequential.mp4|width=640|height=480| path=http://www.s2technologies.com/video/wiki/%7Cmore_than_one=true</dhflashplayer>

Fixed

Fixed baseline comparisons can be used when you want to define a fixed set of data that represents your expected results for each run of the test-bed. The only way to create a fixed baseline currently is by copying it from an existing result set or by importing XML data from an external file -- which would have been either exported from a different test space OR generated by the stride test runner. The snapshot icon () shown in the testspace view allows you to create a baseline from a particular result set. When you select this button, a form will be displayed to allow you to specify a name and optional duration limit percent value to use to calculate duration upper bounds for all test cases. For example, if you enter 20 in this field, the duration upper bound will be calculated by multiplying the duration value for each test case by 1.2 (20% above the nominal value). If you leave this duration field blank when creating a baseline, no upper bound values will be assigned to the baseline. This baseline creation process copies the data in the result set to the fixed baseline - once the baseline is created, any changes to the original result set have no impact on the fixed baseline data.

This screencast shows how to create fixed baselines:

<dhflashplayer>file=TS_BaselineFixedCreate.mp4|width=640|height=480| path=http://www.s2technologies.com/video/wiki/%7Cmore_than_one=true</dhflashplayer>

Fixed baselines can be viewed and edited by going to the baseline tab in your testspace view. The baseline list view supports the following operations for each baseline:

• copy (to a new fixed baseline)
• add data (by uploading an XML file)
• export (to XML file)
• edit (name and description)
• delete

In addition, the individual test cases that make up the baseline data can be modified - specifically the status can be changed and the duration upper bound can be set or modified. These values are modified by first selecting the baseline name in the list view to get to the result set view for the baseline data. Once you have navigated to the desired test case (using hyperlink/tree navigation), you can hover near one of these values (status or duration limit) and select the edit link that appears. Make your changes using the displayed form and submit. When editing the duration limit value, you can enter floating point values representing the upper bound (in milliseconds) OR a percentage string (including the percent sign, e.g. "20%") and the duration limit by applying this percentage increase to the test case's duration value.

This screencast shows how to edit baseline data:

<dhflashplayer>file=TS_BaselineFixedEdit.mp4|width=640|height=480| path=http://www.s2technologies.com/video/wiki/%7Cmore_than_one=true</dhflashplayer>

Once you have creates baseline data, you can configure the testspace to automatically compare new result sets with one or more of the baselines. To do this, edit the testspace properties and select the add fixed baseline compare under the auto compare settings. Using the drop-list of existing fixed baselines, select the desired baseline and submit. You can also remove previously set baseline comparisons (if you no longer want to compare with certain baselines) using the testspace properties page.

This screencast shows how to apply fixed baselines:

<dhflashplayer>file=TS_BaselineFixedApply.mp4|width=640|height=480| path=http://www.s2technologies.com/video/wiki/%7Cmore_than_one=true</dhflashplayer>

How the baseline comparison is computed

When comparing baseline data with actual data, we categorize each difference as either no-change (0 below), progress, or setback. These numbers - progress and setback - are shown whenever there is a comparison of actual data and baseline data. The matrix below outlines what kind of differences contribute to the overall counts.

	pass	fail	in-progress	unknown	missing	not applicable	<<< BASELINE data has
pass	0	progress	progress	progress	0	0
fail	setback	0	setback	setback	0	0
in-progress	setback	0	0	0	0	0
unknown	setback	0	0	0	0	0
missing	setback	setback	setback	setback	0	0
not applicable	0	0	0	0	0	0
^^^^
ACTUAL data has