Training Getting Started: Difference between revisions
Line 203: | Line 203: | ||
=== Publish Test Space Results === | === Publish Test Space Results === | ||
To complete the setup publish your results to Test Space. | To complete the setup publish your results to Test Space. Please make sure to use '''Sandbox''' as the space to upload your results to (see below). | ||
> stride --options_file myoptions.txt --run "*" --space Sandbox --upload | > stride --options_file myoptions.txt --run "*" --space Sandbox --upload |
Revision as of 21:15, 29 January 2013
Overview
Our training approach is based on a self-guided tour of the STRIDE Testing Features using reference examples and assigned exercises. The set of examples and the implemented exercises will be built and executed using a standard desktop computer.
The software under test, reference examples, and exercises have been designed to be as simple as possible while sufficiently demonstrating the topic at hand. In particular, the software under test is very light on core application logic -- the focus instead is on the test code that leverages STRIDE to define and execute tests.
The user is expected to work through each of the training modules covering a specific testing feature. Once the exercises are completed (actual test cases implemented), the results are published to Test Space (and validated against a pre-created baseline).
The training collateral consists of the following:
- The STRIDE Framework used to implement and execute tests
- The Framework is configured in an Off-Target Environment
- A set of specific Training Modules that will guide you through the exercises
- Wiki articles that will provide background material and other technical information
- Reference Samples
- and STRIDE Test Space
For more details concerning STRIDE refer to the following:
- Overview screencast
- What is Unique About STRIDE
- Types of Testing Supported
- Frequently Asked Questions
For questions and support email us
Before Starting
Before you can start setting up your environment you need the following 3 items:
- STRIDE Desktop Installation Package (one of the following)
- STRIDE_framework-windows_4.3.yy.zip (Windows desktop)
or
- STRIDE_framework-linux_4.3.yy.tgz (Linux desktop)
- Training Source files
- STRIDE_training_source.zip
- Test Space User Account URL and Logon Credentials
- Test Space URL (https://YourCompany.stridetestspace.com)
- User-name
- User-password
Desktop Setup
The training requires that you install the Desktop Framework and that the Off-Target Environment is setup correctly and verified.
For an overview of the installation steps please refer to this article.
The following steps are required:
- Install your desktop Framework package
- Read about the Off-Target Environment
- Install host C++ compiler for your desktop (if needed)
- Use the Off-Target SDK to build a Test App
- Run STRIDE diagnostics with the Test App built
Training Setup
The following source code can be found in the STRIDE_training_source.zip file:
software_under_test.c | h TestBasic.cpp | h TestParam.cpp | h TestFixture.cpp | h TestExpect.cpp | h TestRuntime.cpp | h TestFile.cpp | h TestDouble.cpp | h
The software under test is contained in the file software_under_test.c | h. All of the public functions use sut_ as a prefix. All training modules test against this file. Although the test examples are contained in C++ files, most of the test logic is written in standard C.
Once the files are obtained the next step is to rebuild the TestApp using the training source code. Move all of the files into the directory %STRIDE_DIR%\SDK\Windows\sample_src (or $STRIDE_DIR/SDK/Posix/sample_src for Linux) and follow the instructions for Building a TestApp.
Once built, list all Test Units within the generated database file using the following command:
Windows
> stride --database="%STRIDE_DIR%\SDK\Windows\out\TestApp.sidb" --list
Linux
> stride --database="$STRIDE_DIR/SDK/Posix/out/TestApp.sidb" --list
The following remoted Functions and Test Units should be displayed:
Functions sut_strcpy(char const * input, char * output) : void strlen(char const * _Str) : size_t Test Units TestBasic() TestDouble_Reference() TestDouble_Definition() TestExpect_Seq() TestExpect_Data() TestExpect_Misc() TestFile() TestFixture() TestParam(int data1, int data2, char * szString) TestRuntime_Static() TestRuntime_Dynamic(int NumberOfTestCases)
Run Training Tests
Here we will run all tests in the TestApp.sidb database.[1]
- If you haven't done so already, Build TestApp using the SDK makefile
- Invoke TestApp found in the /out/bin directory
- Create an option file (myoptions.txt) that includes the following content (SDK\Windows or SDK/Posix).
Windows
##### Command Line Options ###### --device "TCP:localhost:8000" --database "%STRIDE_DIR%\SDK\Windows\out\TestApp.sidb" --output "%STRIDE_DIR%\SDK\Windows\sample_src\TestApp.xml" --log_level all
Linux
##### Command Line Options ##### --device "TCP:localhost:8000" --database "$STRIDE_DIR/SDK/Posix/out/TestApp.sidb" --output "$STRIDE_DIR/SDK/Posix/sample_src/TestApp.xml" --log_level all
A couple of things to note:
- If you set up an environment variable for the device option then it is not required in the option file. Note: Command line options override environment variables.
- stride --help provides options information
If you haven't done so already, start TestApp running in a separate console window.
Now run stride as shown below and verify summary results (starting from the SDK\Windows or SDK/Posix directory):
> stride --options_file myoptions.txt --run "*"
The output should look like this:
Loading database... Connecting to device... Executing... test unit "TestBasic" > 3 passed, 1 failed, 0 in progress, 1 not in use. test unit "TestDouble_Definition" > 2 passed, 0 failed, 0 in progress, 1 not in use. test unit "TestDouble_Reference" > 2 passed, 0 failed, 0 in progress, 1 not in use. test unit "TestExpect_Data" > 2 passed, 0 failed, 0 in progress, 1 not in use. test unit "TestExpect_Misc" > 3 passed, 1 failed, 0 in progress, 1 not in use. test unit "TestExpect_Seq" > 2 passed, 0 failed, 0 in progress, 1 not in use. test unit "TestFile" > 0 passed, 0 failed, 0 in progress, 1 not in use. test unit "TestFixture" > 1 passed, 0 failed, 0 in progress, 1 not in use. test unit "TestParam" > 0 passed, 1 failed, 0 in progress, 1 not in use. test unit "TestRuntime_Dynamic" > 4 passed, 0 failed, 0 in progress, 1 not in use. test unit "TestRuntime_Static" > 2 passed, 0 failed, 0 in progress, 1 not in use. --------------------------------------------------------------------- Summary: 21 passed, 3 failed, 0 in progress, 11 not in use. Disconnecting from device... Saving result file...
Interpreting Results
Open TestApp.xml in your browser; this file is created in the directory from which you ran stride (or at the path specified via the --output command line option). If you were connected to the Internet when you ran the tests, the TestApp.xsl file is also generated in the directory. By opening TestApp.xml in a web browser, the xsl is automatically applied to create html in the browser. If you use Google Chrome, please see Browser Compatibility.
If you're interested in the details of the tests, please see the test documentation contained in the test report.
Test Space Access
As part of the training, users implement exercises (test cases). As each exercise is completed, the results are expected to be uploaded to Test Space. Accessing Test Space (uploading, viewing, etc.) requires a user-name and password. Before working on a training module please confirm that your user account has been set up by logging in.
Test Space will have expected results using baselines. This is used to automatically verify if the exercises have been implemented correctly (at least to some degree). For capturing test results a Training project has been created with the following Spaces:
Sandbox - Results for training setup TestBasic - Basics training results: TestBasic.cpp | h TestParam - Parameters training results: TestParam.cpp | h TestFixture - Fixturing training results: TestFixture.cpp | h TestExpect - Expectations training results: TestExpect.cpp | h TestRuntime - Runtime API training results: TestRuntime.cpp | h TestFile - File IO training results: TestFile.cpp | h TestDouble - Doubling training results: TestDouble.cpp | h
To publish your results using the STRIDE Runner the following command-line options should be used:
--testspace https://username:password@yourcompany.stridetestspace.com --project Training --space MODULENAME --name YOURNAME
Notes:
- Concerning the MODULENAME option: use the name that corresponds to the training module currently underway (e.g. TestBasic, TestParam, etc.)
- YOURNAME should be set to your name (i.e. JohnD); omit spaces from this string
- If you access the Internet via an HTTP proxy please read this article
To make it easier for now we recommend that you update your existing option file (myoptions.txt) with the following:
#### Test Space options (partial) ##### #### Note - make sure to change username, etc. #### --testspace https://username:password@yourcompany.stridetestspace.com --project Training --name YOURNAME
Publish Test Space Results
To complete the setup publish your results to Test Space. Please make sure to use Sandbox as the space to upload your results to (see below).
> stride --options_file myoptions.txt --run "*" --space Sandbox --upload
Loading database... Connecting to device... Executing... test unit "TestBasic" > 3 passed, 1 failed, 0 in progress, 1 not in use. test unit "TestDouble_Definition" > 2 passed, 0 failed, 0 in progress, 1 not in use. test unit "TestDouble_Reference" > 2 passed, 0 failed, 0 in progress, 1 not in use. test unit "TestExpect_Data" > 2 passed, 0 failed, 0 in progress, 1 not in use. test unit "TestExpect_Misc" > 3 passed, 1 failed, 0 in progress, 1 not in use. test unit "TestExpect_Seq" > 2 passed, 0 failed, 0 in progress, 1 not in use. test unit "TestFile" > 0 passed, 0 failed, 0 in progress, 1 not in use. test unit "TestFixture" > 1 passed, 0 failed, 0 in progress, 1 not in use. test unit "TestParam" > 0 passed, 1 failed, 0 in progress, 1 not in use. test unit "TestRuntime_Dynamic" > 4 passed, 0 failed, 0 in progress, 1 not in use. test unit "TestRuntime_Static" > 2 passed, 0 failed, 0 in progress, 1 not in use. --------------------------------------------------------------------- Summary: 21 passed, 3 failed, 0 in progress, 11 not in use. Disconnecting from device... Saving result file... Uploading to test space...
Confirming Setup
After you have run and uploaded the results of the training setup, you should confirm the correctness of your work. The training setup results will end up with a mix of passes and failures, so we use a Test Space baseline to get an aggregate comparison between expected and actual results.
To validate your setup:
- Navigate to Sandbox set in Test Space and look under the column labeled SEQUENTIAL COMPARISON
- Confirm that none is indicated for each of the sub-column. If so, you have completed the setup and are ready to move onto the training modules.
Training
At this point you should be ready to start the actual training. There are 7 separate training modules and we recommend the following order:
Introductory
- Basics - Covers basics of implementing and executing test cases
- Parameters - How to pass parameters to a test
- Fixturing - Leveraging setup and teardown featuring
Advanced
- Expectations - Validating code sequencing along with state data
- Runtime API - Using the runtime services to dynamically create test suits / cases
- File IO - Reading and writing files existing on the host
- Doubling - Replacing a dependency with a stub, fake, or mock
Training Confirmation
As you run and upload each test unit containing your worked training exercises, you should confirm the correctness of your work.
The correctly worked training test units will end up with a mix of passes and failures, so we use a Test Space baseline to get an aggregate comparison between expected and actual results
To validate your work:
- Navigate to your result set in Test Space and look under the column labeled SEQUENTIAL COMPARISON
- Confirm that none is indicated for each of the sub-column.
Notes
- ↑ Note that the S2 diagnostic tests are treated separately, and are not run unless the --diagnostics option is specified to stride.