Training Test Macros

Objectives

This Training Module focuses on the basics of writing and executing tests. It covers the following topics:

• The different Test Unit packaging options
• How to leverage Test Macros
• Using Notes and Test Logs, what is the difference?
• Creating Test Documentation via the build process
• Executing Tests using the Runner

This training module comprises three separate Test Units, each of which demonstrates an available packaging option:

TestBasic_FList

Packages a group of free functions into a Test Unit

TestBasic_CClass

Uses a C structure as a Test Unit container

TestBasic_Class

Uses a C++ class as a Test Unit container

These test units are implemented in two source files: TestBasic.cpp and TestBasic.h.

All of the test methods in each Test Unit have already been implemented (you can use these for reference) except for the test method namedTestExercise. This method simply sets a NOT IN USE status for the test method.

IMPORTANT NOTE - As you look at this module's source code, you will note that the test methods (cases) within each Test Unit are exactly the same, regardless of packaging. (In fact the CClass and Class Test Units just call the C-based test functions used by the FList Test Unit.) The only difference is in the packaging of the test methods into each Test Unit.

Instructions

Build and Run TestApp

• Build TestApp.exe using the SDK makefile
  • If you have trouble building, please see Troubleshooting Build Problems.
• Invoke TestApp.exe
  • After build, the file will be found in %STRIDE_DIR%\SDK\Windows\out\bin.
  • You can double-click in the File Explorer and the program's console window will open.
• Create an option file (myoptions.txt) that includes the following content (Windows example)
  • You can create this file anywhere that's convenient.

 ##### Command Line Options ######
 --device "TCP:localhost:8000"
 --database %STRIDE_DIR%\SDK\Windows\out\TestApp.sidb
 --output %STRIDE_DIR%\SDK\Windows\sample_src\TestApp.xml
 --log_level all

• Execute the Test Basic Test Units

 > stride -O myoptions.txt --run TestBasic_FList TestBasic_CClass TestBasic_Class

 Loading database...
 Connecting to device...
 Executing...
   test unit "TestBasic_FList"
     > 3 passed, 2 failed, 0 in progress, 1 not in use.
   test unit "TestBasic_CClass"
     > 3 passed, 2 failed, 0 in progress, 1 not in use.
   test unit "TestBasic_Class"
     > 3 passed, 2 failed, 0 in progress, 1 not in use.
   --------------------------------------------------------------
   Summary: 9 passed, 6 failed, 0 in progress, 3 not in use.

 Disconnecting from device...
 Saving result file...

• You can now review the details of the test results using a Web Browser.
• Locate TestApp.xml (which was written to the sample_src directory via the --output command line option).
• Open this file in your browser (dragging the file from file explorer to your browser is usually the easiest way to do this).
  • By opening the xml file in a web browser an xsl style sheet is automatically applied to create html that is ultimately displayed.
  • If you use Google Chrome, please see Browser Compatibility.

Implement Exercises

Now edit the training source code to complete the following exercises:

TestBadFailure()

• Your assignment: The test macro currently used in the test method TestBadFailure() can prevent subsequent test methods in the Test Unit from executing. This is sometimes useful, but here we want to change the Test Macro type that's used such that subsequent test methods will execute.
  • Hint: This function can be found in TestBasic.cpp in the sample_src directory.
  • Hint: You will probably want to read about Test Macros
  • Hint: Reading about Test Units is a good idea too

TestExercise()

• Assignment 1: Add an srNOTE to TestExercise() that will add a simple message to the test report (e.g. "Test Exercise ...")
  • Hint: You will probably want to read about Test Macros

• Assignment 2: Within TestExercise() validate that sut_mult(1,1) does NOT equal sut_add(1,1)
  • Hint: You will use a Test Macro.

Checking Results

• Before you rebuild TestApp.exe, you will need to shut it down by entering 'q' in its console window or closing the window directly.

• After rebuilding, invoke TestApp.exe once again.

• Execute only TestBasic_FList using the stride runner:

 > stride -O myoptions.txt --run TestBasic_FList

 Loading database...
 Connecting to device...
 Executing...
   test unit "TestBasic_FList"
     > 4 passed, 2 failed, 0 in progress, 0 not in use.
   --------------------------------------------------------------
   Summary: 4 passed, 2 failed, 0 in progress, 0 not in use.

 Disconnecting from device...
 Saving result file...

• If you have TestApp.xml already open in your browser, you can simply refresh (F5), to view the latest test results.

Run and Publish Results

When you have completed the Exercise(s) publish your results to Test Space. To make it easier for now we recommend that you update your existing option file (myoptions.txt) with the following:

 #### Test Space options (partial) #####
 #### Note - make sure to change username, etc. ####
 --testspace https://username:password@yourcompany.stridetestspace.com
 --project Training
 --name YOURNAME

  > stride -O myoptions.txt --run TestBasic_FList TestBasic_CClass TestBasic_Class --space TestBasic --upload

 Loading database...
 Connecting to device...
 Executing...
   test unit "TestBasic_FList"
     > 4 passed, 2 failed, 0 in progress, 0 not in use.
   test unit "TestBasic_CClass"
     > 4 passed, 2 failed, 0 in progress, 0 not in use.
   test unit "TestBasic_Class"
     > 4 passed, 2 failed, 0 in progress, 0 not in use.
   ------------------------------------------------------------
   Summary: 12 passed, 6 failed, 0 in progress, 0 not in use.

 Disconnecting from device...
 Saving result file...
 Uploading to test space...

Note: This space has been set up with a Baseline of expected test results that you can use to validate your results.

You can view the results on test space by pointing your browser at https://yourcompany.stridetestspace.com. Log in using the credentials you supplied earlier in the options file.

Reference

The following reference information is related to Test Unit basics.

Wiki

• Test Units Overview - Provides a general overview Test Units (i.e. writing tests in C and C++)
• Test Unit Packaging - Discusses the three types of packaging that can be used for Test Units (we prefer Test Classes even for c programmers)
• Test Macros - Optional macros that provide shortcuts for testing assertions and automatic report annotation (you will want to use these).
• Notes - Used to add logging information to your test logic (automatically added to test reports)
• Test Logs - Used to add logging information to your source code (added to test reports if enabled)
• Tracing - The Runner allows tracing on logs and test points.

Samples

• Test Function List Sample - Specifically the Basic Simple Test Unit
• Test Macro Sample - This sample covers simple uses of each of the Test Macros.

Note - Other Test Unit Samples related to packaging that are useful are the following:

• Test Class Sample
• Test C Class Sample