Training Instrumentation

Background

Source instrumentation is the means by which application domain experts strategically instrument the source under test so as to enable themselves or others to write expectation tests against the running code. Source instrumentation is one of the first steps toward enabling expectation testing of your application.

Source instrumentation is accomplished by a set of simple yet powerful macros provided by the STRIDE Runtime library. The macros are easily activated/deactivated for any build through the use of a single preprocessor macro value. When activated, they provide a means to validate the behavior of your application as it is running.

Please review the following reference articles before proceeding:

• Instrumentation Overview
• Test Points
• Test Logs

How Do I Instrument ?

As described in the overview, you can begin instrumenting your source code by including the srtest.h header file and then adding srTEST_POINT* and srTEST_LOG* in source locations of interest. These macros will be inactive (no-op) unless STRIDE_ENABLED is defined during compilation.

Where Should I Instrument ?

When thinking about where to instrument your source under test, consider these high-value locations:

• Function entry and exit points. Include call parameters as data if appropriate.
• State transitions
• Data transitions (i.e. any point where important data changes value)
• Error conditions

TBD - more about test planning/strategy.

What's The Difference between a Test Point and a Test Log ?

Test Points can be used for validation since they are what's checked when you run a STRIDE expectation test. Test Logs, on the other hand, are purely informational and will be included in the report, according to the log level indicated when the STRIDE Runner was executed. Refer to the background links above for more information on each of these instrumentation types.

What About Data ?

Including data with your test points adds another level of power to the validation of your source. Here are some general recommendations for using data effectively:

• Try to use string data payloads wherever possible. String payloads are considerably more human-friendly when viewing test results and they allow for relatively simple string comparison validation.
• If you need to do complex validation of multi-field data in a test point, consider using an object serialization format such as JSON. Standard formats like this are readily parsable in host scripting languages. If, however, you will only be writing expectation tests in native code for execution on the target, then string serialization formats might be too cumbersome for validation. In that case, using binary payloads (structures, typically) is reasonable.
• TBD
• TBD

Sample Code: s2_expectations_source

For this training, we will simply review some existing sample source code and explain the motivation behind some of the instrumentation. In particular, we will peruse the source code associated with the Expectations Sample.

Samples/test_in_script/Expectations/s2_expectations_source.c

This is the source under test for a simple example that demonstrates expectation where the test logic is written in a script module (perl) that runs on the host. The source under test is a simple state machine and we have chosen to instrument each of the 4 states with one or more test points. Open the source file in your preferred editor and search for srTEST_POINT. You will see that we have test points in the following functions:

• SetNewState(): This is a shared function that is called to effectuate a state change. The test point label is just a static string chosen by the instrumenter and the name of the new state is included as string data on the test point.
• Start(): The Start state includes a single test point with no data. The label is gotten from a shard function that returns a string representation for any given state (GetStateName()).
• Idle(): The Idle state records a single test point. The test point label is again obtained from GetStateName() and the data associated with the test point is a transition count value that the software under test is maintaining. This state function also includes an info level test log. Since it is an info level log, it will not be captured during testing unless you explicitly set the log level to info or higher when executing the tests (via the STRIDE Runner).
• Active(): The Active state has three distinct test points. The first point is similar to previous states and records a test point with the name of the current state as a label and transition count as data. The second test point shows another example of including string data in a test point. The third test point shows an example of simple JSON serialization to include several values in a single test point payload.
• Exp_DoStateChanges(): This function drives the state transitions and includes one warning log message that is not hit during normal execution of the code.
• End(): The End state records a single test point with the state name as label, similar to the other states already mentioned.