Training Test Macros

Objectives

This Training Module focuses on the basics of writing and executing tests. It covers the following topics:

• The different Test Unit packaging options
• How to leverage Test Macros
• Using Notes and Test Logs, what is the difference?
• Creating Test Documentation via the build process
• Executing Tests using the Runner

This training module comprises three separate Test Units, each of which demonstrates an available packaging option:

TestBasic_FList

Packages a group of free functions into a Test Unit

TestBasic_CClass

Uses a C structure as a Test Unit container

TestBasic_Class

Uses a C++ class as a Test Unit container

These test units are implemented in two source files: TestBasic.cpp and TestBasic.h. The comments and descriptions are contained in TestBasic.h.

The Test Unit has test cases already implemented (used for reference) and has test methods that you are required to implement (called TestExercise). This method simply sets a NOT IN USE status for the test method.

IMPORTANT NOTE - As you look at this module's source code, you will note that the test methods (cases) within each Test Unit are exactly the same, regardless of packaging. (In fact the CClass and Class Test Units just call the C-based test functions used by the FList Test Unit.) The only difference is in the packaging of the test methods into each Test Unit.

Instructions

Build and Run TestApp

• Build TestApp using the SDK makefile
• Invoke TestApp found in the /out/bin directory
• Create an option file (myoptions.txt) that includes the following content in either (SDK\Windows or SDK/Posix).

Windows

 ##### Command Line Options ######
 --device "TCP:localhost:8000"
 --database "%STRIDE_DIR%\SDK\Windows\out\TestApp.sidb"
 --output "%STRIDE_DIR%\SDK\Windows\sample_src\TestApp.xml"
 --log_level all

Linux

 ##### Command Line Options #####
 --device "TCP:localhost:8000"
 --database "$STRIDE_DIR/SDK/Posix/out/TestApp.sidb"
 --output "$STRIDE_DIR/SDK/Posix/sample_src/TestApp.xml"
 --log_level all 

• Execute the Test Basic Test Units

 > stride --options_file myoptions.txt --run TestBasic_FList --run TestBasic_CClass --run TestBasic_Class

 Loading database...
 Connecting to device...
 Executing...
   test unit "TestBasic_FList"
     > 3 passed, 2 failed, 0 in progress, 1 not in use.
   test unit "TestBasic_CClass"
     > 3 passed, 2 failed, 0 in progress, 1 not in use.
   test unit "TestBasic_Class"
     > 3 passed, 2 failed, 0 in progress, 1 not in use.
   --------------------------------------------------------------
   Summary: 9 passed, 6 failed, 0 in progress, 3 not in use.

 Disconnecting from device...
 Saving result file...

You can also review the details of the test results using a Browser. Open TestApp.xml which can be found in the sample_src directory (based on the output option). By opening the xml file in a web browser the xsl is automatically applied to create html.

Implement Exercises

Now edit the training source code to complete the following exercises:

TestBadFailure()

• Your assignment: The test macro currently used in the test method TestBadFailure() can prevent subsequent test methods in the Test Unit from executing. This is sometimes useful, but here we want to change the Test Macro type that's used such that subsequent test methods will execute.
  • Hint: This function can be found in TestBasic.cpp in the sample_src directory.
  • Hint: You will probably want to read about Test Macros
  • Hint: Reading about Test Units is a good idea too

TestExercise()

• Assignment 1: Add an srNOTE to TestExercise() that will add a simple message to the test report (e.g. "Test Exercise ...")
  • Hint: You will probably want to read about Test Macros

• Assignment 2: Within TestExercise() validate that sut_mult(1,1) does NOT equal sut_add(1,1)
  • Hint: You will use a Test Macro.

Checking Results

• Before you rebuild TestApp.exe, you will need to shut it down by entering 'q' in its console window or closing the window directly.

• After rebuilding, invoke TestApp.exe once again.

• Execute only TestBasic_FList using the stride runner:

 > stride --options_file myoptions.txt --run TestBasic_FList

 Loading database...
 Connecting to device...
 Executing...
   test unit "TestBasic_FList"
     > 4 passed, 2 failed, 0 in progress, 0 not in use.
   --------------------------------------------------------------
   Summary: 4 passed, 2 failed, 0 in progress, 0 not in use.

 Disconnecting from device...
 Saving result file...

• If you have TestApp.xml already open in your browser, you can simply refresh (F5), to view the latest test results.

Run and Publish Results

When you have completed the Exercise(s) publish your results to Test Space. To make it easier for now we recommend that you update your existing option file (myoptions.txt) with the following:

 #### Test Space options (partial) #####
 #### Note - make sure to change username, etc. ####
 --testspace https://username:password@yourcompany.stridetestspace.com
 --project Training
 --name YOURNAME

  > stride --options_file myoptions.txt --run TestBasic_FList --run TestBasic_CClass --run TestBasic_Class --space TestBasic --upload

 Loading database...
 Connecting to device...
 Executing...
   test unit "TestBasic_FList"
     > 4 passed, 2 failed, 0 in progress, 0 not in use.
   test unit "TestBasic_CClass"
     > 4 passed, 2 failed, 0 in progress, 0 not in use.
   test unit "TestBasic_Class"
     > 4 passed, 2 failed, 0 in progress, 0 not in use.
   ------------------------------------------------------------
   Summary: 12 passed, 6 failed, 0 in progress, 0 not in use.

 Disconnecting from device...
 Saving result file...
 Uploading to test space...

Validate Uploaded Results

Navigate to Test Space using your browser, then validate your results against the pre-configured baseline. For details, please see Confirming Training Exercise Results

Note: This space has been set up with a Baseline of expected test results that you can use to validate your results.

You can view the results on test space by pointing your browser at https://yourcompany.stridetestspace.com. Log in using the credentials you supplied earlier in the options file.

Reference

The following reference information is related to Test Unit basics.

Wiki

• Test Units Overview - Provides a general overview Test Units (i.e. writing tests in C and C++)
• Test Unit Packaging - Discusses the three types of packaging that can be used for Test Units (we prefer Test Classes even for c programmers)
• Test Macros - Optional macros that provide shortcuts for testing assertions and automatic report annotation (you will want to use these).
• Notes - Used to add logging information to your test logic (automatically added to test reports)
• Test Logs - Used to add logging information to your source code (added to test reports if enabled)
• Tracing - The Runner allows tracing on logs and test points.

Samples

• Test Function List Sample - Specifically the Basic Simple Test Unit
• Test Macro Sample - This sample covers simple uses of each of the Test Macros.

Note - Other Test Unit Samples related to packaging that are useful are the following:

• Test Class Sample
• Test C Class Sample