STRIDE Wiki - User contributions [en]

Expectations Sample

2010-07-13T23:06:03Z

Mikee: /* Tests Description */

== Introduction ==
This example demonstrates a simple technique to monitor and test activity occurring in instrumented source code on the device from test logic implemented on the host. This sample show a common testing scenario - namely, verifying the behavior of a state machine.

If you are not familiar with test points you may find it helpful to review the [[Test Point]] article before proceeding.

== Source under test ==

=== <tt>s2_expectations_source.c / h</tt> ===
These files implement a simple state machine that we wish to test. The state machine runs when '''Exp_DoStateChanges''' is executed, which is a function we have also instrumented so it can be remotely invoked from the host.

The expected state transitions are as follows:

<pre>
eSTART -> eIDLE -> eACTIVE -> eIDLE -> eEND
</pre>

The states don't do any work; instead they just sleep() so there's some time spent in each one.

Each state transition is managed through a call to SetNewState() which communicates the state transition to the test thread using the srTEST_POINT() macro. We also provide an incrementing counter value as data to each of these test points - this data is used for validation in one of our example scenarios. The source under test has also been instrumented with a test point to illustrate the use of JSON as a textual serialization format (which is easily decoded on the host).

== Tests Description ==

=== s2_expectations_testmodule ===

This example implements four tests of the state machine implemented in s2_expectations_source. These tests demonstrate the use of the [[Perl Script APIs]] to validate expectations.

Each test follows the same pattern in preparing and using the test point feature:
# call '''TestPointSetup''' with order parameter as well as expected and unexpected lists.
# Invoke target processing by calling the remote function '''Exp_DoStateChanges'''.
# use '''Check''' or '''Wait''' on the test point object to process the expectations.

We create an "expectation" of activity and then validate the observed activity against the expectation using rules that we specify. If the expectation is met, the test passes; if the expectation is not met, the test fails.

==== sync_exact ====

This is test implements a basic '''ordered''' and '''strict''' expectation test (ordered/strict
processing is the default behavior for ''TestPointSetup'', so there is no need to explicitly
specify these settings). An ordered/strict test expects the declared events to occur in
the order specified and with no other intervening events from among those already declared in
the list.

The test strictness normally applies to the universe of declared test points - that is, the union
of all points declared in the expected and unexpected lists. However, for this example, we want to
ensure that '''no other''' test points are encountered during the test. As such, we specify an
unexpected list containing '''TEST_POINT_EVERYTHING_ELSE''' which makes this check entirely exclusive
on the expected list.

==== sync_loose ====
This test relaxes some of the restrictions of the previous test by
specifying '''unordered''' processing AND removing the '''unexpected''' list.
As a result, this test simply validates that the declared test points
occur with the specified frequency during the processing. This test ''does not''
validate the ordering of the events nor does it exclude other events that are
outside the expectation universe (that is, test points NOT mentioned in the expectation
list)

Note that the '''IDLE''' testpoint is now included in the expected array only once, but with an
expected count of 2. This technique is common when using unordered processing.

This test will fail only if all of the expected testpoints
are not seen (the specified number of times) during the processing window.

==== async_loose ====
This test is identical to the sync_loose test, except that we call '''Wait()'''
and pass a timeout value to 200 milliseconds, which will result in a test failure, as it
takes approximately 600 milliseconds for the testpoint expectations to be satisfied
(the source under test includes artificial delays between each state transition).

'''This test is expected to fail'''

==== check_data ====
This test is identical to sync_loose, except that we use '''ordered''' processing
for an ordered expectation set '''and''' we specify expected data for some of our test points. This
test will pass only if the test points are seen in the specified order and match both the label
and data specified.

For the test points with binary data, we have to use the perl [http://perldoc.perl.org/functions/pack.html pack()] function to create a scalar
value that has the proper bit pattern. Whenever you are validating target data, you will need to
take into account byte ordering for basic types. Here we assume the target has the same byte
ordering as the host. A more scalable approach to data validation from the target involves
using string-based serialization formats (such as JSON).

==== trace_data ====
This test is similar to sync_exact, except the expectations are loaded from a
expected data file that was created using the <tt>--trace</tt> option on the [[STRIDE Runner]].
We've also removed the unexpected list so the items are the trace file are not
considered exclusively.

By default, tests that use a expect data file perform validation based only on
the test point label. If you want data comparison to be performed, you need to
specify a global predicate via the '''predicate''' option.

==== trace_data_validation ====

This test is identical to ''trace_data'', but with the addition of data validation
using the provided [[Perl_Script_APIs#Builtin_Predicates|TestPointDefaultCmp predicate]].

==== trace_data_custom_predicate ====
This test is identical to ''trace_data'' except that a custom predicate
is specified for the data validation. The custom predicate in this example
just validates binary data using the standard memory comparison and implicitly
passes for any non-binary payloads.

==== json_data ====
This test demonstrates the use of [http://www.json.org JSON] formatted string data from the target in
predicate validation. The string payload for the test point is decoded
using the standard perl JSON library and the object's fields are validated
in the predicate function above.

==== non_strict_ordered ====
This test demonstrates the use of non-strict processing which allows
other occurrences of test points within your current universe. In this test,
there are other occurrences of the '''SET_NEW_STATE''' event that we have not
explicitly specified in our expectation list. Had we been using strict
processing (as is the default), these extra occurrences would have caused
the test to fail. Because we have specified non-strict processing, the
test passes.

==== unexpected ====
This test demonstrates the use of the unexpected list to ensure
that specific test points are not hit during the check. In this case,
we specify JSON_DATA in our unexpected list - however, since this
test points ''does'' actually occur during our processing, the test fails.

'''This test is expected to fail'''

==== startup_predicate ====
This test demonstrates the use of the special '''TEST_POINT_ANYTHING'''
specification to implement startup logic. Sometimes it is desirable
to suspend processing of your expectation list until a certain event
occurs. Startup predicate logic allows you to accomplish this, as this
example demonstrates. In this somewhat contrived example, we run our event
scenario twice consecutively. The startup logic waits for the '''END''' event
to occur. Once it does, the predicate returns true and the remaining items
are processed in turn.

==== anything ====
Similar to the '''startup_predicate''' test, this example demonstrates
the use of '''TEST_POINT_ANYTHING''' in ''the middle'' of an expectation list
to suspend further processing of the list until a condition is satisfied.
The proceed condition is enforced by a predicate, this any expectation
entries using TEST_POINT_ANYTHING '''must''' include a predicate to validate
the condition. In this example, the TEST_POINT_ANYTHING suspends processing
of the list until it sees the '''IDLE''' event.

==== predicate_validation ====
This test shows that you can use a custom predicate along with
'''TEST_POINT_ANY_AT_ALL'''. In this way, you can force all validation
to be handled in your predicate. The predicate should return
0 to fail the test, 1 to pass the test, or '''TEST_POINT_IGNORE''' to
continue processing other test points.

This technique should only be used in cases where the other
more explicit expectation list techniques are not sufficient.

==== multiple_handlers ====

This is test shows how you can use multiple test point setups simultaneously with each
processing a different set of expectations. This can be a powerful technique and
often produces more understandable test scenarios, particularly when you decompose
your expetations down into small, easy to digest expectation lists.

In this specific case, we create one handler that expects START and END, in
sequence and another that just verifies that the IDLE event happens twice. We
deliberately create both handlers before starting the processing so that
each handler will have access to the stream of events as they happen during
processing.

[[Category:Samples]]

Expectations

2010-07-13T22:25:24Z

Mikee: /* Members */

== Overview ==

STRIDE supports '''Behavior Testing''', which is different than Unit Testing or API Testing in that it does not focus on calling functions and validating return values. Behavior Testing validates the ''expected sequencing and state of the software'' executing under normal operating conditions. In order to begin this type of testing, you must '''(1)''' instrument the source under test with [[Test Point | Test Points]] and '''(2)''' define the ''expectations'' of the Test Points for each test scenario. To learn more about the uniqueness of Behavior Testing [[What_is_Unique_About_STRIDE#STRIDE_includes_behavior-based_testing_techniques | read here]].

The '''Validation Model''' is based on validating that the ''Expected List'' of Test Points have been hit as well as optionally validating any ''state data'' associated with a Test Point member. Through the use of an ''Unexpected List'', it's possible to simulataneously validate that specified Test Points are NOT hit during the execution of the scenario.

The ''Expected List'' and ''Unexpected List'' declared for a specific testing scenario define the ''active'' set of Test Points for the system. All other Test Points within the software are not active and have nominal impact. To learn more about testable software builds [[What_is_Unique_About_STRIDE#STRIDE_software_builds_are_both_functional_and_testable | read here]].

== Expected List ==

The first step in defining '''Expectations''' is describing the set of Test Points expected to be hit during a test scenario. This is the '''list''' of Test Points and the expected number of hits ('''count''') for each of the Test Points. Any expected '''state data''' associated with a Test Point that requires validation should be included.

The validation of the ''Expected List'' is done when any of the following conditions are met:
* The ''Expected List'' sequencing has been met completely
* The time allocated for validation has expired
* State data associated with a Test Point is declared invalid
* An out-of-sequence Test Point has been encountered (a concrete failure is identified)
* An [[Expectations#Unexpected_List | Unexpected Test Point]] has been encountered

'''Expected TEST POINTS list:'''

{| border="1" cellspacing="0" cellpadding="10" style="align:left;"
| '''Label'''
| '''Count'''
| '''Expected State Data'''
|-
| Name 1
| 1
| ''<describe data payload validation requirements if applicable>''
|-
| Name 2
| 1 +
| ''<describe data payload validation requirements if applicable>''
|-
| ''name ...''
| ''n''
| ''<...>''
|}

=== Sequencing Properties ===

The ''Expected List'' requires defining ''sequencing properties'' used for the validation. This involves establishing how to process the '''ordering''' of the Test Points being hit and how '''strict''' to handle duplications of Test Point members.

'''Expected TEST POINTS sequencing properties:'''

{| border="1" cellspacing="0" cellpadding="10" style="align:left;"
| '''Properties'''
| ''' Description'''
|-
| ''Ordered''
| Test Points are expected to be hit in the exact order defined in the list
|-
| ''Unordered''
| Test Points can be hit in any order defined in the list
|-
| ''Strict''
| Test Points specified in the list must match the exact count (i.e. no duplication)
|-
| ''Non-Strict''
| Test Points specified in the list can be duplicated anywhere in the sequence
|}

=== State Data Validation ===

State data validation for a Test Point member is optional. The ''Expected List'' can associate a ''predicate''<ref name="predicate">Defined as a necessary condition of the Test Point being valid -- logic within the predicate determines if ''valid'', ''invalid'', or ''ignored'' </ref> as a necessary condition for validation of a Test Point member when being processed. The predicate provides an extra level of validation (beyond sequencing) focusing on the state data associated with the Test Point.

The predicate can declare the Test Point:
* Valid - successful, thus the expected count is decremented
* Invalid - the expectations is NOT met, indicating failure
* Ignored - no affect on the validation, continue to wait on the current Test Point

== Unexpected List ==

An ''Unexpected List'' of Test Points can optionally be defined. This list works in conjunction with the [[Expectations#Expected_List | ''Expected List'']]. The list defines a set of Test Points that are to be treated as failures if any one of them is encountered (hit) during a testing scenario.

This list can be made up of the '' '''complement''' '' of the ''Expected List'', thus including ''all'' Test Points not included in that defined set. This type of setting requires enabling all Test Points defined in the software during the execution of the test.

If the ''Unexpected List'' members intersect with the ''Expected List'' members, that is considered an input error.

== Special Processing ==

=== Members ===
There is one ''special member'' that can be used for the ''Expected List'' called '''ANYTHING'''. This special member value is used to customize the success criteria (i.e. trigger condition) for the entry defined within the ''Expected List''. It requires a predicate<ref name="predicate"/> to determine if the Test Point is ''valid, invalid, or ignored''. The '''ANYTHING''' refers to the set of points defined by the ''Expected List''.

Concerning an '''Unordered''' ''Expected List'', only one special member can be used. It will be processed first independent of its location within the list. The '''Count''' attribute is also optionally available to be used with the special members, indicating the number of valid returns from the predicate until the next Test Point in the list is validated.

There is one ''special member'' that can be used for the ''Unexpected List'' called '''EVERYTHING ELSE'''. This special member can only be used as a '''single member only''' within an ''Unexpected List''. It takes the ''complement'' of the set of Test Points defined within the ''Expected List''. This requires enabling all Test Points defined in the software during the execution of the test. Also, by definition any Test Point hit during the test would be deemed a failure.

=== Counts ===
There is one ''special count'' value that can be associated with a Test Point defined in the ''Expected List'' called '''ANY COUNT'''. This special count is used to focus on capturing a variable number of Test Point hits -- 0 or more. In essences the Test Point has no consequences of the Expected List sequencing, except if a predicate<ref name="predicate"/> associated with the Test Point returns ''invalid''.

== Use Cases ==

The following use cases provide examples of defining '''Expectations''' based on different testing scenarios.

=== Sequencing Properties ===
Use case examples for validating the [[Expectations#Sequencing_Properties | sequencing]] of Test Points.

Expecting an exact ordered sequence of Test Points to be hit
''Expect'' = [A,B,C,D]; ''ORDERED, STRICT''
Hits = [A,B,C,D] - PASS
Hits = [A,B,D,C] - FAIL (D hit before C)

Expecting an exact ordered of Test Points but with duplicates
''Expect'' = [A,B,C,D]; ''ORDERED, STRICT''
Hits = [A,B,B,C,D] - FAIL (B duplicated with strict defined)

''Expect'' = [A,B,C,D]; ''ORDERED, NON-STRICT''
Hits = [A,B,B,C,D] - PASS
Hits = [A,C,B,C,D] - PASS
Hits = [A,B,C,B,D] - PASS

Expecting a set of Test Points to be hit, but the order does not matter
''Expect'' = [A,B,C,D]; ''UNORDERED, STRICT''
Hits = [B,A,C,D] - PASS
Hits = [B,A,C,B,D] - FAIL (B hit twice)

''Expect'' = [A,B,C,D]; ''UNORDERED, NON-STRICT''
Hits = [B,A,C,B,D] - PASS

Expecting a set of Test Points to be hit, but order and duplications does not matter
''Expect'' = [A,B,C,D]; ''UNORDERED, NON-STRICT''
Hits = [B,A,C,B,D] - PASS
Hits = [B,A,A,A,D] - FAIL (C never hit)

=== Count Attribute ===
Use case examples that leverage the [[Expectations#Count | Count]] attribute associated with a list member.

Expecting a sequence of Test Points, including multiple hits for 2 members
''Expect'' = [A,B:3,C,D:2]; ''ORDERED, STRICT''
Hits = [A,B,B,B,C,D,D] - PASS
Hits = [A,B,B,C,B,D,D] - FAIL (B not hit three times before C)

''Expect'' = [A,B:3,C,D:2]; ''ORDERED, NON-STRICT''
Hits = [A,B,B,B,C,B,D,D] - PASS

Expecting a set of Test Points being hit, including multiple hits (with duplications), but order does not matter
''Expect'' = [A,B:3,C,D]; ''UNORDERED, STRICT''
Hits = [B,A,B,C,B,D] - PASS
Hits = [B,A,B,B,C,B,D] - FAIL (B duplicated on 4th hit)

''Expect'' = [A,B:3,C,D]; ''UNORDERED, NON-STRICT''
Hits = [B,A,B,B,C,B,D] - PASS

=== State Data Validation ===
Use case examples leveraging predicates<ref name="predicate"/> to [[Expectations#State_Data_Validation | validate state data]].

Simple example showing associating a predicate with Test Point named ''A''.
''Expect'' = [A:p1(tp),B,C,D]; ''ORDERED, STRICT''
p1(tp) { return VALID }
Hits = [A,B,C,D] - PASS

Expecting Test Point ''B'' to be hit 3 times
''Expect'' = [A,B:3:p1(tp),C,D]; ''UNORDERED, STRICT''
p1(tp) { return VALID unless 3rd time called than return INVALID}
Hits = [B,A,B,C,B,D] - FAIL (predicate returns invalid on 3rd B hit)

=== Special Members ===
Use case examples using the two [[Expectations#Members | special members]] within an ''Expected List''.

Waiting on a single trigger (D) to start the processing of an Expected List
''Expect'' = [ANYTHING:p1(tp),A,B,C:2,D]; ''ORDERED, STRICT''
p1(tp) {return VALID when D hit otherwise IGNORE}
Hits = [A,B,A,D*,A,B,C,C,D] - PASS

Waiting on a multiple triggers - 1st trigger used to start processing and 2nd used midstream
''Expect'' = [ANYTHING:p1(tp),A,B,ANYTHING:p2(tp),C:2,D,E]; ''ORDERED, STRICT''
p1(tp) {return VALID when D hit}
p2(tp) {return VALID when E hit}
Hits = [A,B,A,D*,A,B,A,A,B,E*,C,C,D,E] - PASS

''Expect'' = [ANY_IN_SET:p1(tp),A,B,C:2,D]; ''UNORDERED, NON-STRICT''
p1(tp) {return VALID when D hit}
Hits = [A,B,A,D*,A,A,A,D,A,C,B,C,D] - PASS

=== Special Counts ===
Use case examples leveraging the [[Expectations#Special_Counts | special count attributes]] within an ''Expected List''.

Expecting between 0 and more Test Point As to be hit before processing the remaining members.
''Expect'' = [A:ANY_COUNT,B,C,D]; ''ORDERED, STRICT''
Hits = [B,C,D] - PASS
Hits = [A,B,C,D] - PASS

This test passes ''immediately'' after the A is hit (example of a weird expectation)
''Expect'' = [A,B:ANY_COUNT]; ''ORDERED, STRICT''
Hits = [A,..] - PASS

=== Miscellaneous ===
The following are miscellaneous use cases.

Expecting an exact ordered sequence of Test Points to be hit, but verifying certain Test Points are NOT hit
''Expect'' = [A,B,C,D]; ''ORDERED, STRICT''
''Unexpect'' = [x,y]
Hits = [A,B,C,D] - PASS
Hits = [A,B,B,C,D] - FAIL
Hits = [A,B,C,y,..] - FAIL

== Notes ==
<references/>

[[Category: Source Instrumentation]]

Script Samples

2010-07-09T22:34:03Z

Mikee:

These samples are a collection of native source under test and script test code code that demonstrate the techniques for creating and executing tests using the STRIDE Framework. These are the introductory samples for the STRIDE Framework that illustrate how to do expectation testing of instrumented source under test using script test modules on the host.

Once you have installed the STRIDE framework on your host machine, you can easily build and run any combination of these samples. In each case, you can include the source under test for the sample simply by copying its source files to the SDK's [[Framework_Installation#SDK | sample_src]] directory and rebuilding the off-target testapp.

= Description =
The following samples are available to illustrate the use of host scripting with the STRIDE Framework.

;[[Expectations Sample]]
: This sample provides some pre-instrumented source code that include [[Test Point|STRIDE Test Points]] and [[Test Log|Test Logs]]. A single perl test module is included that implements a few examples of expectation testing based on the software under test. The process of including this sample as well as running it and publishing results is covered in [[Running and Publishing the Expectations Sample]], which comprised an earlier step in the sandbox setup.

;[[FileTransfer Sample]]
: This sample shows an example of how you might use helper functions on the target to invoke [[File Transfer Services|STRIDE File Transfer services]]. The example is driven by host script logic that invokes remote target functions to actuate a file transfer to the target.

;[[FunctionRemoting Sample]]
: This sample shows some examples of invoking [[Function_Capturing|remote functions]] for the purpose of fixturing your behavior tests written in script on the host.

[[Category:Samples]]
[[Category:Tests in Script]]

FunctionRemoting Sample

2010-07-09T22:30:14Z

Mikee: Created page with '== Introduction == This sample demonstrates some common remote function call patterns for the purpose of fixturing your script test scenarios. This sample only demonstrates the …'

== Introduction ==

This sample demonstrates some common remote function call patterns for the purpose of fixturing your script test scenarios. This sample only demonstrates the STRIDE mechanics of qualifying and invoking the remote functions. The example functions here don't actually implement any fixturing logic - they only verify the parameters passed and returned. This sample is a good place to start if you are interested in creating some functions to provided on-target fixturing for your host-based behavior tests.

== Source under test ==

=== <tt>s2_function_remoting_source.c / h</tt> ===

These files declare and implement several example functions taking and returning a variety of input/output data. In particular, we have examples of:
* void functions
* integer input/return
* string input/return
* structure input/return
* variable sized integer array input/return
* variable sized string array input/return

== Tests Description ==

=== void_args_void_return ===

demonstrates calling a function with void args and return value.

=== int_args_int_return ===

demonstrates calling a function with integer arg and return value.

=== string_args_string_return ===

demonstrates calling a function with string arg and return value.

=== struct_args_struct_return ===

demonstrates calling a function with struct arg and return value.

=== intsizedarray_args_intsizedarray_return ===

demonstrates calling a function with variable length integer array and return value. When using sized-pointers, you must assign the size field with correct number of elements in the array. The return value is a struct which gets mapped to a hashref in perl.

=== stringsizedarray_args_stringsizedarray_return ===

demonstrates calling a function with variable length string array and return value. When using sized-pointers, you must assign the size field with correct number of elements in the array. The return value is a struct which gets mapped to a hashref in perl.

[[Category:Samples]]

Frequently Asked Questions About STRIDE

2010-06-16T19:26:49Z

Mikee: /* Testing */

== Integration with STRIDE ==

=== What is the size of the STRIDE Runtime? ===

The [[STRIDE Runtime]] is a source package that supports connectivity with the host system and provides[[Runtime_Test_Services | services for testing]] and [[Source_Instrumentation_Overview | source instrumentation]]. The ''runtime'' is tailored specifically to embedded applications, overhead is minimal. It consumes very little memory for table and control block storage. Resource usage is [[STRIDE_Runtime#Runtime_Configuration | configurable]] and can be tailored to the limitations of the target platform.

{| class="wikitable"
|+ Typical resource usage
! Aspect !! Resources
|-
| Code Space || About 90-130 KB depending on the compiler of use and the level of optimization.
|-
| Memory Usage || Configurable, by default set to about 10 KB
|-
| Threads || 3 Threads; configurable priority; blocked when inactive
|}

=== What is the processing overhead? ===

The [[STRIDE Runtime]] overhead is minimized by collecting raw data on the target, and transferring information at a low priority task to the host for processing. Processing is only active when executing tests via the [[STRIDE Runner]].

== Source Instrumentation ==

=== What are the advantages of Test Points over logging? ===

[[Test Point | Test Points]] can be used for direct validation during real-time execution. Logging systems, while often useful for historical record, can only be used for post-processing validation at best. What's more, since there is no standard way to post-process logs, users often rely on manual inspection or non-scalable homegrown solutions for validation. Test Point validation is fully automated using the STRIDE Framework, which provides robust harnessing and reporting. Test Points also can include optional data payloads which can be very useful for validating state when a point is hit. Test Points have potentially lower impact on the system than standard logging since they '''(1)''' are only sent from the origin if a test is actively subscribed to test point (labels are used as a filter at runtime) and '''(2)''' test points can be completely disabled (no-opped) in your build by undefining a single preprocessor macro [[Runtime_Integration#STRIDE_Feature_Control | (STRIDE_ENABLED)]].

=== What about source instrumentation bloat? ===

Any mature code base has some level of diagnostic bloat to it. This often takes the form of ad-hoc logging or debug statements. With STRIDE [[Test Point | Test Points]] and [[Test Log | Test Logs]], you open your software to better automated test scenarios. All STRIDE instrumentation takes the form of single line macros, so the amount of instrumentation bloat is no worse than other typical ad-hoc diagnostics. What's more, the STRIDE macros are all designed to be quickly no-opped in a build via a single preprocessor macro, making it possible to completely eliminate any actual impact on certain builds, if so desired.

=== Are all Test Points active? ===

No. Under normal testing scenarios, only the specific test points that are needed for a test are actually broadcast through the system. We accomplish this by setting test point filters (by label) on the system whenever one of the Test Point setup functions is called (in script or native code). These filters are reset or removed at the end of the test, so in general ''none'' of the test points are actually sent through the system if no test is currently active. That said, there are a few special use cases in which ''all'' test points become active in the system - namely, when [[Tracing | tracing]] is activated on the host runner and when a specific test case uses a set that include ''TEST_POINT_EVERYTHING_ELSE''. In general, however, the test points that are actually sent from the system are ''only'' those that are needed to execute the behavior validation for the current test.

=== Will it affect performance? ===

Our experience on a wide-range of systems has shown minimal impact from the STRIDE instrumentation. The STRIDE Runtime has been designed to be small, portable, and readily configurable, allowing it to be optimized for the platform's specific characteristics.

=== Should I leave Test Points in? ===

Yes. Once you have some behavior tests written, it's worthwhile to maintain that instrumentation and the corresponding tests, which allows you to run the tests on any stride-enabled build. All of the instrumentation macros are easily no-opped via a single preprocessor flag, so you can choose to effectively remove the instrumentation code on select builds (production/release builds, for example). The ultimate value of instrumentation is the continuous feedback you get by regularly executing the automated tests on the build.

== Testing ==

=== What languages are supported ? ===

[[Types_of_Testing_Supported#Unit Testing | Unit tests]] and [[Types_of_Testing_Supported#API_Testing | API Tests]] are written using '''C and C++'''. [[Types_of_Testing_Supported#Behavior_Testing | Behavior tests]] can be written using '''Perl''' and '''C/C++'''.

=== Is there any alternative to running STRIDE tests with a real device ? ===

Yes. Tests can be also be built and executed using the [[Off-Target Environment]]. In order for this to work, your device source code must be built along with the test code using the host's desktop toolchain (MSVC on windows, gcc on linux).

== Test Automation ==

=== What is continuous integration and why should I care? ===

The key principle of continuous integration is regular testing of your software--ideally done in an automated fashion. STRIDE tests are reusable and automated. Over time, these tests accumulate, providing more and more comprehensive coverage. By automating the execution of tests and results publication via [[STRIDE Test Space]] with every software build, development teams gain immediate feedback on defects and the health of their software. By detecting and repairing defects immediately, the expense and time involved with correcting bugs is minimized.

=== Does STRIDE support continuous integration? ===

Yes. The [[STRIDE Runner]] provides a straightforward means to connect to the device under test and execute the test cases you've implemented using the STRIDE Framework. The runner allows you to configure which tests to run and how to organize the results using subsuites in the report. Since the runner supports an option-based command line interface, this tool is easy to integrate with typical continuous integration servers. Please refer to [[Setting_up_your_CI_Environment|this article]] as well.

=== Where/how do you store test results? ===

When you execute your tests using the [[STRIDE Runner]], upon completion the results are written to an xml file. This xml file uses a custom schema for representing the hierarchy of results (suites, cases, etc.). These files also include a stylsheet specification (which will be written to the same directory as the xml file) that allows them to be viewed as HTML in a browser. You are free to store these files for future use/reference. The runner '''also''' supports direct uploading to [[STRIDE Test Space]] which is a hosted web application for viewing and collaborating on your test results. Once you are regularly executing tests, whether automatically or manually, we recommend you use the upload feature to persist and share your test results.

=== Can I get email containing test reports? ===

Yes. If you use [[STRIDE Test Space]] to store your results, you can optionally configure your test space(s) to automatically notify users when new results are uploaded. The email that is generated by Test Space contains only summary information and provides links so that you can view the complete report data.

If you are using a continuous integration server to initiate your testing, it's likely that it supports different forms of notification when the testing is complete, so it's often possible to attach the xml report data as part of the CI server notification.

== Getting Started ==

=== How long does it take to install STRIDE? ===

For standard platforms such as Linux the installation process varies between a few hours to a couple of days. We provide [[:Category:SDKs|SDK packages]] that work out of the box for our [[Off-Target Environment]], but also provide a reference to integrators.

For proprietary targets a [[Platform_Abstraction_Layer|Platform Abstraction Layer (PAL)]] is required. The PAL provides the glue between the [[STRIDE Runtime]] and services offered by the OS. It is the only piece of the runtime that is customized between operating systems. The implementation of the PAL ranges between a day to a week depending on the complexity of the OS. There is also a [[Build Integration | build integration]] step. This involves integrating the [[STRIDE Build Tools]] into your software make process. The activity ranges from a single day to several.

=== What kind of training is required? ===

Our training [[Training Overview | approach]] is based on wiki articles, [[Samples | samples]], and leveraging the [[Off-Target Environment]]. The training have been setup for self-guided instructions that can be leverage for an initial introduction of the technology and on-demand for specific topics when required.

[[Category: Overview]]

Frequently Asked Questions About STRIDE

2010-06-16T17:39:58Z

Mikee: /* Are all Test Points active? */

== Integration with STRIDE ==

=== What is the size of the STRIDE Runtime? ===

The [[STRIDE Runtime]] is a source package that supports connectivity with the host system and provides[[Runtime_Test_Services | services for testing]] and [[Source_Instrumentation_Overview | source instrumentation]]. The ''runtime'' is tailored specifically to embedded applications, overhead is minimal. It consumes very little memory for table and control block storage. Resource usage is [[STRIDE_Runtime#Runtime_Configuration | configurable]] and can be tailored to the limitations of the target platform.

{| class="wikitable"
|+ Typical resource usage
! Aspect !! Resources
|-
| Code Space || About 90-130 KB depending on the compiler of use and the level of optimization.
|-
| Memory Usage || Configurable, by default set to about 10 KB
|-
| Threads || 3 Threads; configurable priority; blocked when inactive
|}

=== What is the processing overhead? ===

The [[STRIDE Runtime]] overhead is minimized by collecting raw data on the target, and transferring information at a low priority task to the host for processing. Processing is only active when executing tests via the [[STRIDE Runner]].

== Source Instrumentation ==

=== What are the advantages of Test Points over logging? ===

[[Test Point | Test Points]] can be used for direct validation during real-time execution. Logging systems, while often useful for historical record, can only be used for post-processing validation at best. What's more, since there is no standard way to post-process logs, users often rely on manual inspection or non-scalable homegrown solutions for validation. Test Point validation is fully automated using the STRIDE Framework, which provides robust harnessing and reporting. Test Points also can include optional data payloads which can be very useful for validating state when a point is hit. Test Points have potentially lower impact on the system than standard logging since they '''(1)''' are only sent from the origin if a test is actively subscribed to test point (labels are used as a filter at runtime) and '''(2)''' test points can be completely disabled (no-opped) in your build by undefining a single preprocessor macro [[Runtime_Integration#STRIDE_Feature_Control | (STRIDE_ENABLED)]].

=== What about source instrumentation bloat? ===

Any mature code base has some level of diagnostic bloat to it. This often takes the form of ad-hoc logging or debug statements. With STRIDE [[Test Point | Test Points]] and [[Test Log | Test Logs]], you open your software to better automated test scenarios. All STRIDE instrumentation takes the form of single line macros, so the amount of instrumentation bloat is no worse than other typical ad-hoc diagnostics. What's more, the STRIDE macros are all designed to be quickly no-opped in a build via a single preprocessor macro, making it possible to completely eliminate any actual impact on certain builds, if so desired.

=== Are all Test Points active? ===

No. Under normal testing scenarios, only the specific test points that are needed for a test are actually broadcast through the system. We accomplish this by setting test point filters (by label) on the system whenever one of the Test Point setup functions is called (in script or native code). These filters are reset or removed at the end of the test, so in general ''none'' of the test points are actually sent through the system if no test is currently active. That said, there are a few special use cases in which ''all'' test points become active in the system - namely, when [[Tracing | tracing]] is activated on the host runner and when a specific test case uses a set that include ''TEST_POINT_EVERYTHING_ELSE''. In general, however, the test points that are actually sent from the system are ''only'' those that are needed to execute the behavior validation for the current test.

=== Will it affect performance? ===

Our experience on a wide-range of systems has shown minimal impact from the STRIDE instrumentation. The STRIDE Runtime has been designed to be small, portable, and readily configurable, allowing it to be optimized for the platform's specific characteristics.

=== Should I leave Test Points in? ===

Yes. Once you have some behavior tests written, it's worthwhile to maintain that instrumentation and the corresponding tests, which allows you to run the tests on any stride-enabled build. All of the instrumentation macros are easily no-opped via a single preprocessor flag, so you can choose to effectively remove the instrumentation code on select builds (production/release builds, for example). The ultimate value of instrumentation is the continuous feedback you get by regularly executing the automated tests on the build.

== Testing ==

=== What languages are supported? ===

[[Types_of_Testing_Supported#Unit Testing | Unit tests]] and [[Types_of_Testing_Supported#API_Testing | API Tests]] are written using '''C and C++'''. [[Types_of_Testing_Supported#Behavior_Testing | Behavior tests]] can be written using '''Perl''' and '''C/C++'''.

=== Can I test using my host computer? ===

Yes. Tests can be implemented and executing using the [[Off-Target Environment]].

== Test Automation ==

=== What is continuous integration and why should I care? ===

The key principle of continuous integration is regular testing of your software--ideally done in an automated fashion. STRIDE tests are reusable and automated. Over time, these tests accumulate, providing more and more comprehensive coverage. By automating the execution of tests and results publication via [[STRIDE Test Space]] with every software build, development teams gain immediate feedback on defects and the health of their software. By detecting and repairing defects immediately, the expense and time involved with correcting bugs is minimized.

=== Does STRIDE support continuous integration? ===

Yes. The [[STRIDE Runner]] provides a straightforward means to connect to the device under test and execute the test cases you've implemented using the STRIDE Framework. The runner allows you to configure which tests to run and how to organize the results using subsuites in the report. Since the runner supports an option-based command line interface, this tool is easy to integrate with typical continuous integration servers. Please refer to [[Setting_up_your_CI_Environment|this article]] as well.

=== Where/how do you store test results? ===

When you execute your tests using the [[STRIDE Runner]], upon completion the results are written to an xml file. This xml file uses a custom schema for representing the hierarchy of results (suites, cases, etc.). These files also include a stylsheet specification (which will be written to the same directory as the xml file) that allows them to be viewed as HTML in a browser. You are free to store these files for future use/reference. The runner '''also''' supports direct uploading to [[STRIDE Test Space]] which is a hosted web application for viewing and collaborating on your test results. Once you are regularly executing tests, whether automatically or manually, we recommend you use the upload feature to persist and share your test results.

=== Can I get email containing test reports? ===

Yes. If you use [[STRIDE Test Space]] to store your results, you can optionally configure your test space(s) to automatically notify users when new results are uploaded. The email that is generated by Test Space contains only summary information and provides links so that you can view the complete report data.

If you are using a continuous integration server to initiate your testing, it's likely that it supports different forms of notification when the testing is complete, so it's often possible to attach the xml report data as part of the CI server notification.

== Getting Started ==

=== How long does it take to install STRIDE? ===

For standard platforms such as Linux the installation process varies between a few hours to a couple of days. We provide [[:Category:SDKs|SDK packages]] that work out of the box for our [[Off-Target Environment]], but also provide a reference to integrators.

For proprietary targets a [[Platform_Abstraction_Layer|Platform Abstraction Layer (PAL)]] is required. The PAL provides the glue between the [[STRIDE Runtime]] and services offered by the OS. It is the only piece of the runtime that is customized between operating systems. The implementation of the PAL ranges between a day to a week depending on the complexity of the OS. There is also a [[Build Integration | build integration]] step. This involves integrating the [[STRIDE Build Tools]] into your software make process. The activity ranges from a single day to several.

=== What kind of training is required? ===

Our training [[Training Overview | approach]] is based on wiki articles, [[Samples | samples]], and leveraging the [[Off-Target Environment]]. The training have been setup for self-guided instructions that can be leverage for an initial introduction of the technology and on-demand for specific topics when required.

[[Category: Overview]]

Frequently Asked Questions About STRIDE

2010-06-16T17:38:44Z

Mikee: /* What are the advantages of Test Points over logging? */

== Integration with STRIDE ==

=== What is the size of the STRIDE Runtime? ===

The [[STRIDE Runtime]] is a source package that supports connectivity with the host system and provides[[Runtime_Test_Services | services for testing]] and [[Source_Instrumentation_Overview | source instrumentation]]. The ''runtime'' is tailored specifically to embedded applications, overhead is minimal. It consumes very little memory for table and control block storage. Resource usage is [[STRIDE_Runtime#Runtime_Configuration | configurable]] and can be tailored to the limitations of the target platform.

{| class="wikitable"
|+ Typical resource usage
! Aspect !! Resources
|-
| Code Space || About 90-130 KB depending on the compiler of use and the level of optimization.
|-
| Memory Usage || Configurable, by default set to about 10 KB
|-
| Threads || 3 Threads; configurable priority; blocked when inactive
|}

=== What is the processing overhead? ===

The [[STRIDE Runtime]] overhead is minimized by collecting raw data on the target, and transferring information at a low priority task to the host for processing. Processing is only active when executing tests via the [[STRIDE Runner]].

== Source Instrumentation ==

=== What are the advantages of Test Points over logging? ===

[[Test Point | Test Points]] can be used for direct validation during real-time execution. Logging systems, while often useful for historical record, can only be used for post-processing validation at best. What's more, since there is no standard way to post-process logs, users often rely on manual inspection or non-scalable homegrown solutions for validation. Test Point validation is fully automated using the STRIDE Framework, which provides robust harnessing and reporting. Test Points also can include optional data payloads which can be very useful for validating state when a point is hit. Test Points have potentially lower impact on the system than standard logging since they '''(1)''' are only sent from the origin if a test is actively subscribed to test point (labels are used as a filter at runtime) and '''(2)''' test points can be completely disabled (no-opped) in your build by undefining a single preprocessor macro [[Runtime_Integration#STRIDE_Feature_Control | (STRIDE_ENABLED)]].

=== What about source instrumentation bloat? ===

Any mature code base has some level of diagnostic bloat to it. This often takes the form of ad-hoc logging or debug statements. With STRIDE [[Test Point | Test Points]] and [[Test Log | Test Logs]], you open your software to better automated test scenarios. All STRIDE instrumentation takes the form of single line macros, so the amount of instrumentation bloat is no worse than other typical ad-hoc diagnostics. What's more, the STRIDE macros are all designed to be quickly no-opped in a build via a single preprocessor macro, making it possible to completely eliminate any actual impact on certain builds, if so desired.

=== Are all Test Points active? ===

No. Under normal testing scenarios, only the specific test points that are needed for a test are actually broadcast through the system. We accomplish this by setting test point filters (by label) on the system whenever one of the Test Point setup functions is called (in script or native code). These filters are reset or removed at the end of the test, so in general ''none'' of the test points are actually sent through the system if no test is currently active. That said, there are a few special use cases in which ''all'' test points become active in the system - namely, when [[Tracing | tracing]] is activated on the host runner and when a specific test case uses a set that include ''TEST_POINT_ANYTHING'' or ''TEST_POINT_EVERYTHING_ELSE''. In general, however, the test points that are actually sent from the system are ''only'' those that are needed to execute the behavior validation for the current test.

=== Will it affect performance? ===

Our experience on a wide-range of systems has shown minimal impact from the STRIDE instrumentation. The STRIDE Runtime has been designed to be small, portable, and readily configurable, allowing it to be optimized for the platform's specific characteristics.

=== Should I leave Test Points in? ===

Yes. Once you have some behavior tests written, it's worthwhile to maintain that instrumentation and the corresponding tests, which allows you to run the tests on any stride-enabled build. All of the instrumentation macros are easily no-opped via a single preprocessor flag, so you can choose to effectively remove the instrumentation code on select builds (production/release builds, for example). The ultimate value of instrumentation is the continuous feedback you get by regularly executing the automated tests on the build.

== Testing ==

=== What languages are supported? ===

[[Types_of_Testing_Supported#Unit Testing | Unit tests]] and [[Types_of_Testing_Supported#API_Testing | API Tests]] are written using '''C and C++'''. [[Types_of_Testing_Supported#Behavior_Testing | Behavior tests]] can be written using '''Perl''' and '''C/C++'''.

=== Can I test using my host computer? ===

Yes. Tests can be implemented and executing using the [[Off-Target Environment]].

== Test Automation ==

=== What is continuous integration and why should I care? ===

The key principle of continuous integration is regular testing of your software--ideally done in an automated fashion. STRIDE tests are reusable and automated. Over time, these tests accumulate, providing more and more comprehensive coverage. By automating the execution of tests and results publication via [[STRIDE Test Space]] with every software build, development teams gain immediate feedback on defects and the health of their software. By detecting and repairing defects immediately, the expense and time involved with correcting bugs is minimized.

=== Does STRIDE support continuous integration? ===

Yes. The [[STRIDE Runner]] provides a straightforward means to connect to the device under test and execute the test cases you've implemented using the STRIDE Framework. The runner allows you to configure which tests to run and how to organize the results using subsuites in the report. Since the runner supports an option-based command line interface, this tool is easy to integrate with typical continuous integration servers. Please refer to [[Setting_up_your_CI_Environment|this article]] as well.

=== Where/how do you store test results? ===

When you execute your tests using the [[STRIDE Runner]], upon completion the results are written to an xml file. This xml file uses a custom schema for representing the hierarchy of results (suites, cases, etc.). These files also include a stylsheet specification (which will be written to the same directory as the xml file) that allows them to be viewed as HTML in a browser. You are free to store these files for future use/reference. The runner '''also''' supports direct uploading to [[STRIDE Test Space]] which is a hosted web application for viewing and collaborating on your test results. Once you are regularly executing tests, whether automatically or manually, we recommend you use the upload feature to persist and share your test results.

=== Can I get email containing test reports? ===

Yes. If you use [[STRIDE Test Space]] to store your results, you can optionally configure your test space(s) to automatically notify users when new results are uploaded. The email that is generated by Test Space contains only summary information and provides links so that you can view the complete report data.

If you are using a continuous integration server to initiate your testing, it's likely that it supports different forms of notification when the testing is complete, so it's often possible to attach the xml report data as part of the CI server notification.

== Getting Started ==

=== How long does it take to install STRIDE? ===

For standard platforms such as Linux the installation process varies between a few hours to a couple of days. We provide [[:Category:SDKs|SDK packages]] that work out of the box for our [[Off-Target Environment]], but also provide a reference to integrators.

For proprietary targets a [[Platform_Abstraction_Layer|Platform Abstraction Layer (PAL)]] is required. The PAL provides the glue between the [[STRIDE Runtime]] and services offered by the OS. It is the only piece of the runtime that is customized between operating systems. The implementation of the PAL ranges between a day to a week depending on the complexity of the OS. There is also a [[Build Integration | build integration]] step. This involves integrating the [[STRIDE Build Tools]] into your software make process. The activity ranges from a single day to several.

=== What kind of training is required? ===

Our training [[Training Overview | approach]] is based on wiki articles, [[Samples | samples]], and leveraging the [[Off-Target Environment]]. The training have been setup for self-guided instructions that can be leverage for an initial introduction of the technology and on-demand for specific topics when required.

[[Category: Overview]]

Perl Script APIs

2010-06-14T21:00:47Z

Mikee:

The STRIDE Framework perl script model requires you to create perl modules (*.pm files) to group your tests. The following documents the API for the STRIDE::Test base class that you use when creating these modules.

== STRIDE::Test ==

This is the base class for test modules. It provides the following methods.

=== Declaring Tests ===

Once you have created a package that inherits from STRIDE::Test, you can declare any subroutine to be a test method by declaring it with the '': Test'' attribute. In addition to test methods, the following attributes declare other kinds of subroutines:

{| class="prettytable"
| colspan="2" | '''subroutine attributes'''

|-valign="top"
! attribute!! description

|-valign="top"
| Test
| declares a test method - will be executed automatically when the module is run.

|-valign="top"
| Test(startup)
| startup method, called once before any of the test methods have been executed.

|-valign="top"
| Test(shutdown)
| shutdown method, called once after all test methods have been executed.

|-valign="top"
| Test(setup)
| setup fixture, called before each test method.

|-valign="top"
| Test(teardown)
| teardown fixture, called after each test method.

|}

You are free to declare as many methods as you like with these attributes. When more than one method has been declared with the same attribute, the methods will be called at the appropriate time in the order declared.

=== Methods ===

These methods are all available in the context of a test module that inherits from STRIDE::Test.

{| class="prettytable"
| colspan="2" | '''Methods'''

|-valign="top"
! name !! description
|-valign="top"
| <source lang="perl">TestCase</source>
| returns the current test case object.

|-valign="top"
| <source lang="perl">TestSuite</source>
| returns the current test suite object.

|-valign="top"
| <source lang="perl">AddAnnotation(
test_case => case,
label => "",
level => LEVEL,
message => "")</source>
| Adds an annotation to the current test case (or to test_case if that named parameter is provided). A ''label'' can be provided but will default to "Host Annotation". A ''level'' can be provided as one of the following, but will default to INFO level:
* '''ANNOTATION_TRACE'''<ref name="const">symbols like these are exported as perl constants - don't quote these values when you use them - rather, use the bare symbols and perl will use the constant value we provide</ref>
* '''ANNOTATION_DEBUG'''
* '''ANNOTATION_INFO'''
* '''ANNOTATION_WARNING'''
* '''ANNOTATION_ERROR'''
* '''ANNOTATION_FATAL'''
The ''message'' parameter is optional and specifies the text to use in the annotation description field.

|-valign="top"
| <source lang="perl">AddTestCase(suite)</source>
| Creates a new test case in the specified suite (or the current TestSuite(), if none is provided). This also updates the current test case value that is returned by TestCase().

|-valign="top"
| <source lang="perl">AddTestSuite(suite)</source>
| Creates a new sub-suite in the specified suite (or the current TestSuite(), if none is provided). This also updates the current test suite value that is returned by TestSuite().

|-valign="top"
| <source lang="perl">Functions</source>
| Returns a [[#STRIDE::Function|STRIDE::Function]] object that was initialized with the active database. This object is used for calling captured functions in the database.

|-valign="top"
| <source lang="perl">Constants</source>
| Returns a [[#STRIDE::Function|STRIDE::Function->{constants}]] hash that was initialized with the active database. This object is a tied hash that can be used to retrieve database constant values (macros and enums) at the time of compilation.

|-valign="top"
| <source lang="perl">TestPointSetup(
expected => [],
unexpected => [],
ordered => 0|1,
strict => 0|1,
expect_file => filepath,
predicate => coderef,
replay_file => filepath,
test_case => case)</source>
| Creates a new instance of [[#STRIDE::TestPoint|STRIDE::TestPoint]], automatically passing the default TestCase() as the test case if none is provided. Options are passed using hash-style arguments. The supported arguments are:
; ordered : flag that specifies whether or not the expectation set must occur in the order specified. If set to true, the expectation list is treated as an ordered list, otherwise we assume the list is unordered. Default value is true (ordered processing).
; strict : flag that specifies if the list is exclusive. If strict is specified, then the actual events (within the specified universe of test points) must match the expectation list exactly as specified. If strict processing is disabled, then other test points within the universe are allowed to occur between items specified in the expectation list. The default value is true (strict processing).
; expected : is an array reference containing elements that are either strings representing the test point labels '''OR''' an anonymous sub-array that contains these four items: '''[label, count, predicate, expected_data]'''. The four element arrayref elements are:
* ''label'' is the test point label (same as the non array argument type). the label is also allowed to be specified using the predefined constant value '''TEST_POINT_ANYTHING'''<ref name="const"/> to indicate a place in the expectation list where '''any''' test point within the current universe is permitted. When this special value is used in the expectation list, a predicate must ''also'' be specified. The test point is only considered satisfied when the predicate returns true. '''TEST_POINT_ANYTHING''' can be used to implement, among other things, startup scenarios where you want to defer your expectation list processing until a particular test point and/or data state have been encountered. When specifying '''TEST_POINT_ANYTHING''' in an '''unordered''' expectation, it is only allowed to appear '''once''' in the expectation list and, in that case, it is treated as a startup expectation whereby none of the other test points are processed until the '''TEST_POINT_ANYTHING''' expectation has been satisfied (when it's predicate returns true).
* ''count'' is the number of expected occurrences - can be any positive integer value, or the special value '''TEST_POINT_ANY_COUNT'''<ref name="const"/>.
* ''predicate'' must be a perl coderef for a function to call as a predicate. You are free to define your own predicate function OR use on the three [[#Predicates | standard ones]] provided by STRIDE::Test.
* ''expected_data'' will be passed as an argument to the predicate - intended to be used to specify expected data.
If using the array form of expectation, only the label entry is required - the remaining elements are optional.
; unexpected : is an array reference containing labels that are to be treated as failure if they are encountered. For either ''expected'' and ''unexpected'', the special value '''TEST_POINT_EVERYTHING_ELSE'''<ref name="const"/> can be used alone to indicate that any test points not explicitly listed in the set are considered part of this set.
; expect_file : if you have previously captured trace data in a file (using the [[STRIDE Runner]]), you can specify the file as the source of expectations using this parameter. If you specify a trace file, you should NOT also specify the '''expected''' items as they will be overridden by the trace file.
; predicate : is a perl coderef to a default predicate function to use for all expectation items. If any specific entry in the expectation list has a predicate function, the expectation's predicate will override this global value. By default, no global predicate is assumed and no predicate is called unless specified for each expectation item.
; replay_file : allows you to specify a trace file and '''''input''''' to the current test. If specified, the expectation will be validated against the events specified in the file rather than live events generated by a device under test.
; test_case : allows you to specify a test case to use for reporting the results. This is only useful for advanced users that are generating test cases dynamically within a test method.

The returned object is of type [[#STRIDE::TestPoint|STRIDE::TestPoint]] and has access to all it's member functions.

|}

=== Assertions ===

Each of the following assertion methods are provided for standard comparisons. For each, there there are three different types, depending on the desired behavior upon failure: '''EXPECT''', '''ASSERT''', and '''EXIT'''. '''EXPECT''' checks will fail the current test case but continue executing the test method. '''ASSERT''' checks will fail the current test case and exit the test method immediately. '''EXIT''' checks will fail the current test case, immediately exit the current test method AND cease further execution of the test module.

{| class="prettytable"
| colspan="2" | '''Boolean'''

|-
! macro !! Pass if
|-
| '''''prefix'''''_TRUE(''cond'');
| ''cond'' is true

|-
| '''''prefix'''''_FALSE(''cond'');
| ''cond'' is false

|}

{| class="prettytable"
| colspan="2" | '''Comparison'''
|-
! macro !! Pass if

|-
| '''''prefix'''''_EQ(''val1'', ''val2'');
| ''val1'' == ''val2''

|-
| '''''prefix'''''_NE(''val1'', ''val2'');
| ''val1'' != ''val2''

|-
| '''''prefix'''''_LT(''val1'', ''val2'');
| ''val1''<nowiki> < </nowiki>''val2''

|-
| '''''prefix'''''_LE(''val1'', ''val2'');
| ''val1''<nowiki> <= </nowiki>''val2''

|-
| '''''prefix'''''_GT(''val1'', ''val2'');
| ''val1'' > ''val2''

|-
| '''''prefix'''''_GE(''val1'', ''val2'');
| ''val1'' >= ''val2''

|}

For all of the value comparison methods ('''_EQ''', '''_NE''', etc.), the comparison is numeric if both arguments are numeric -- otherwise the comparison is a case sensitive string comparison. If case insensitive comparison is needed, simply wrap both arguments with perl's builtin '''lc()''' (lowercase) or '''uc()''' (uppercase) functions.

{| class="prettytable"
| colspan="2" | '''Predicates'''
|-
! macro !! Pass if

|-
| '''''prefix'''''_PRED(''coderef'', ''data'')
| ''&coderef''(''data'') returns true. The predicate function is specified by ''coderef'' with optional data ''data''. The predicate can also return the special value '''TEST_POINT_IGNORE'''<ref name="const"/>' to indicate that the event should be ignored.

|}

Each of these expectation methods also supports the following optional named arguments:
; test_case => case : allows you to apply the check to a test case other than the current default
; message => "message" : allows you to specify an additional message to include if the check fails.

Because these arguments are optional, they are passed using named argument (hash-style) syntax after the required parameters that are shown above.

=== Logging ===

The following methods can be used to annotate a test case. Typically these methods are used to add additional information about the state of the test to the report.

{| class="prettytable"
| colspan="2" | '''Logging Methods'''

|-valign="top"
! name !! description

|-valign="top"
| <source lang="perl">NOTE_INFO(message)</source>
| creates an info note in your test results report.

|-valign="top"
| <source lang="perl">NOTE_WARN(message)</source>
| creates an warning note in your test results report.

|-valign="top"
| <source lang="perl">NOTE_ERROR(message)</source>
| creates an error note in your test results report.

|}

Each of these note methods also supports the following optional named arguments:
; test_case => case : allows you to add the log to a test case other than the current default
; file => file : allows you to attach a file along with the annotation message that is generated for the log message.
; test_point => test_point_hashref : If you are annotating your report in the context of a predicate with a specific test point, you might want to specify the test point using this parameter. This will cause your annotation to be grouped in the final report with the annotation message that corresponds to the test point hit message. By default, a host timestamp value is used to generate the NOTE annotation, which generally causes the NOTE annotations to group toward the end of the test case report.

Because these arguments are optional, they are passed using named argument (hash-style) syntax after the required parameters that are shown above.

=== Documentation ===

We have preliminary support for documentation extraction in the test modules using the standard perl POD formatting tokens.

The POD that you include in your test module currently must follow these conventions:

* it must begin with a ''head1 NAME'' section and the text of this section must contain the name of the package, preferably near the beginning.
* a ''head1 DESCRIPTION'' can follow the ''NAME'' section. If provided, it will be used as the description of the test suite created for the test unit.
* This NAME/DESCRIPTION block must finish with an empty ''head1 METHODS'' section.
* each of the test methods can be documented by preceding them with a ''head2'' section with the same name as the test method (subroutine name). The text in this section will be used as the testcase description.

=== Predicates ===

STRIDE expectation testing allows you to specify predicate functions for sophisticated data validation. We provide several standard predicates in the STRIDE::Test package, or you are free to define your own predicate functions.

==== Builtin Predicates ====
The STRIDE::Test library provides a few standard predicates which you are free to use in your expectations:

{| class="prettytable"
| colspan="2" | '''Built-In Predicates'''

|-valign="top"
! predicate !! description

|-valign="top"
| '''TestPointStrCmp'''
| does a case '''''sensitive''''' comparison of the test point data and the ''expected_data'' (specified as part of the expectation)

|-valign="top"
| '''TestPointStrCaseCmp'''
| does a case '''''insensitive''''' comparison of the test point data and the ''expected_data''

|-valign="top"
| '''TestPointMemCmp'''
| does a bytewise comparison of the test point data and the ''expected_data''

|-valign="top"
| '''TestPointDefaultCmp'''
| pass-through function that calls '''TestPointMemCmp''' for binary test point data or '''TestPointStrCmp''' otherwise. This is useful as a global predicate since it implements an appropriate default data comparison.

|}

==== User Defined Predicates ====
User defined predicates are subroutines of the following form:

<source lang="perl">
sub myPredicate
{
my ($test_point, $expected_data) = @_;
my $status = 0;
# access the test point data as $test_point->{data},
# and the label as $test_point->{label}

# set $status according to whether or not your predicate passes
return $status;
}
</source>

The predicate function is passed two arguments: the current test point and the expected data that was specified as part of the expectation. The test point data is a reference to a hash with the following fields:
; label : the test point label
; data : the data payload for the test point (if any)
; data_as_hex : an alternate form of the data payload, rendered as a string of hex characters
; size : the size of the data payload
; bin : flag indicating whether or not the data payload is binary
; file : the source file for the test point
; line : the line number for the test point

The expected data is passed as a single scalar, but you can use references to compound data structures (hashes, arrays) if you need more complex expected data.

The predicate function should return a true value if it passes, false if not, or '''TEST_POINT_IGNORE'''<ref name="const"/> if the test point should be ignored completely.

== STRIDE::Function ==

The STRIDE::Function class uses perl '''AUTOLOAD'''ing to provide a convenient syntax for making simple function calls and retrieving database constants in perl. Given any properly initialized Function object, any captured function or constant (macro) is available directly as method or properties of the Function object. Constants can also be accessed via the tied hash ''constants'' member (which is exported as '''Constants''' in the STRIDE::Test package).

For example, given a database with two functions: '''int foo(const char * path)''' and '''void bar(double value)''', and a macro '''#define MY_PI_VALUE 3.1415927''', these methods/constants are invokable using the exported STRIDE::Test Functions and Constants objects:

<source lang="perl">
my $retval = Functions->foo("my string");
Functions->bar(Constants->{MY_PI_VALUE});
</source>

=== Asynchronous invocation ===

Functions can also be called asynchronously by accessing functions using the {async} delegator within the function object. When invoked this way, the function call will return a handle object that can be used to wait for the function return value - for example:

<source lang="perl">
my $h = Functions->{async}->foo("my string");
my $retval = $h->Wait(1000);
</source>

The '''Wait''' function takes one optional argument -- the timeout duration (in milliseconds) that indicates the maximum time to wait for the function to return. If the timeout value is not provided, ''Wait'' will wait indefinitely for the function to return. If a timeout is specified and expires before the function returns, the method with ''die'' with a timeout error message - so you might want to wrap your '''Wait''' call in an <tt>eval{};</tt> statement if you want to gracefully handle the timeout condition.

== STRIDE::TestPoint ==

STRIDE::TestPoint objects are used to create test point expectation tests. These objects are created using the exported TestPointSetup factory function of the STRIDE::Test class. Once a STRIDE::TestPoint object has been created with the desired expectations, two functions can be called:

; Wait(timeout) : This method processes test points that have occurred on the target and assesses failure based on the parameters you provided when creating the TestPoint object. The timeout parameter indicates how long (in milliseconds) to Wait for the specified events. If no timeout value is provided, '''Wait''' will proceed indefinitely or until a clear pass/failure determination can be made.
; Check() : this is equivalent to Wait with a very small timeout. As such, it essentially verifies that your specified test points have already been hit.

== Notes ==
<references/>

[[Category:Tests in Script]]

Source Instrumentation Overview

2010-06-11T23:51:07Z

Mikee:

== Introduction ==
Source instrumentation is the process by which developers and domain experts selectively instrument the source under test for the purpose of writing test scenarios against the executing application. Implementing tests that leverage source instrumentation is called '''Expectation Testing'''. This validation technique is very useful for verifying proper code sequencing based on the software's internal design.

Unlike traditional unit testing that drives testing based on input parameters and isolating functionality, expectation testing is executed within a fully functional software build running on a real target platform. Expectation tests are not dependent on input parameters, but often leverage the same types of input/output controls used by functional and black-box testing.

Another unique feature of expectation testing is that domain expertise is not required to implement a test. Developers and domain experts use instrumentation to export design knowledge of the software to the entire team. Furthermore, there is no stubbing required, no special logic to generate input parameters, and no advanced knowledge required of how the application software is coded.

To enable effective test coverage, developers and domain experts are required to insert instrumentation at key locations to gain insight and testability. Here are some general suggested source code areas to consider instrumenting:
* critical function entry/exit points
* state transitions
* critical or interesting data transitions (using optional payload to convey data values)
* callback routines
* data persistence
* error conditions

== Instrumentation ==
To make the software ''testable'', the first step in the process is for the experts to selectively insert instrumentation macros -- called [[Test_Point | Test Points]] -- into the source code. Test Points themselves have nominal impact on the performance of the application – they are only active during test data collection<ref name="n1"> Test data collection is typically implemented in a low priority background thread. The data is captured in the calling routine's thread context (no context switch) but processed in the background or on the host. Instrumentation macros return immediately to the caller (i.e. NOP)</ref> when testing is not active. Test Points contain names and optional payload data. When Test Points are activated, they are collected in the background, along with timing and any associated data. The set of Test Points hit, their order, timing, and data content can all be used to validate that the software is behaving as expected. [[Test_Log | Test Logs]] can also be added to the source code to provide additional information in the context of an executing test.

Here are the requirements for source instrumentation using the STRIDE Framework:

* define the '''STRIDE_ENABLED''' preprocessor macro in your build system
* include the '''srtest.h''' header file. This file is included in the [[Distribution_Files#STRIDE_Runtime | Runtime source distribution]]
* selectively instrument strategic locations in your source code with [[Test_Point | Test Points]] and optionally [[Test_Log | Test Logs]]

== Expectations ==

In addition to instrumenting the source code with Test Points, you must also define the [[Expectations]] of the Test Points. This involves defining the list of Test Points expected to be hit during a given test scenario. An expectation can also include any expected '''data''' associated with a Test Point that requires validation.

== Testing ==

Once the source under test has been instrumented and the [[Expectations | expectations]] defined, STRIDE offers a number of techniques that can be used for implementing '''expectation tests'''. For non-developers the [[Test_Modules_Overview | STRIDE Scripting Solution]] is recommended. Scripting allows testers to leverage the power of dynamic languages that execute on the host. The framework provides script libraries that automate the behavior validation as well as hooks for customization. The script implementation also reduces software build dependencies since the test code is not part of the device image. Scripting can also leverage [[Function_Capturing | function remoting]] to fully automate test execution by [[Perl_Script_Snippets#Invoking_a_function_on_the_target | invoking functions on the target.]]

For developers writing expectation tests, the [[Test_Units_Overview | STRIDE Test Units]] with test logic implemented in [[Expectation_Tests_in_C/C%2B%2B | C or C++]] is recommended as a starting point.

== Notes ==
<references/>

[[Category: Source Instrumentation]]

Source Instrumentation Overview

2010-06-11T23:43:08Z

Mikee: /* Introduction */

== Introduction ==
Source instrumentation is the process by which developers and domain experts selectively instrument the source under test for the purpose of writing test scenarios against the executing application. Implementing tests that leverage source instrumentation is called '''Expectation Testing'''. This validation technique is very useful for verifying proper code sequencing based on the software's internal design.

Unlike traditional unit testing that drives testing based on input parameters and isolating functionality, expectation testing is executed within a fully functional software build running on a real target platform. Expectation tests are not dependent on input parameters, but often leverage the same types of input/output controls used by functional and black-box testing.

Another unique feature of expectation testing is that domain expertise is not required to implement a test. Developers and domain experts use instrumentation to export design knowledge of the software to the entire team. Furthermore, there is no stubbing required, no special logic to generate input parameters, and no advanced knowledge required of how the application software is coded.

To enable effective test coverage, developers and domain experts are required to insert instrumentation at key locations to gain insight and testability. Here are some general suggested source code areas to consider instrumenting:
* critical function entry/exit points
* state transitions
* critical or interesting data transitions (using optional payload to convey data values)
* callback routines
* data persistence
* error conditions

== Instrumentation ==
To make the software ''testable'' the first step is the process is for the experts to selectively insert instrumentation macros called [[Test_Point | Test Point]] into the source code. Test Points themselves have nominal impact on the performance of the application – they are only active during test data collection<ref name="n1"> Test data collection is typically implemented in a low priority background thread. The data is captured in the calling routine's thread context (no context switch) but processed in the background. When testing is not active instrumentation macros return immediately to the caller (i.e. NOP)</ref>. Test Points contains names and optionally payload data. When Test Points are activated, they are collected in the background, along with timing and any associated data. The set of Test Points hit, their order, timing, and data content can all be used to validate the software is behaving as expected. [[Test_Log | Test Logs]] can also be added to the source code to provide additional information in the context of an executing test.

Here are the requirements for source instrumentation using the STRIDE Framework:

* define the '''STRIDE_ENABLED''' preprocessor macro in your build system
* include the '''srtest.h''' header file. This file is included in the [[Distribution_Files#STRIDE_Runtime | Runtime source distribution]]
* selectively instrument strategic locations in your source code with [[Test_Point | Test Points]] and optionally [[Test_Log | Test Logs]]

== Expectations ==
In addition to instrumenting the source code with Test Points, defining the [[Expectations]] of the Test Points is required for testing. The involves defining the list of Test Points expected to be hit during a given test scenario. Also included is any expected '''data''' associated with a Test Point that requires validation.

== Testings ==
Once the source under test has been instrumented and the [[Expectations | expectations]] defined, STRIDE offers a number of techniques to be used for implementing '''expectation tests'''. For non-developers the [[Test_Modules_Overview | STRIDE Scripting Solution]] is recommended. Scripting allows testers to leverage the power of dynamic languages that execute on the host. The framework provides script libraries that automate the behavior validation as well as hooks for customization. Also there are minimal software build dependencies using scripting for validation. Scripting can also leverage [[Function_Capturing | function remoting]] to fully automate test execution using script modules [[Perl_Script_Snippets#Invoking_a_function_on_the_target | invoking functions on the target.]]

For developers writing expectation tests the [[Test_Units_Overview | STRIDE Test Units]] and implementation in [[Expectation_Tests_in_C/C%2B%2B | C or C++]] is recommended to begin with.

== Notes ==
<references/>

[[Category: Source Instrumentation]]

What is Unique About STRIDE

2010-06-11T23:36:16Z

Mikee: /* STRIDE optimizes failure resolution */

== STRIDE is a cross-platform framework ==
The [[Runtime_Reference | '''STRIDE Runtime''']] is written in standard C on top of a simple [[Platform_Abstraction_Layer | '''platform abstraction layer''']] that enables it to work on virtually any target platform. It is delivered as source code to be included in the application's build system. STRIDE also auto-generates [[Intercept_Module | '''harnessing and remoting logic''']] as source code during the make process, removing any dependencies on specific compilers and / or processors. The transport between the host and target is configurable and supports serial/USB and TCP/IP by default. Custom transports are also available. STRIDE is easily configured for single process or multiple process environments (i.e. Linux, Windows CE, etc.). Testing can also be conducted using an [[Off-Target_Environment | '''off-target environment''']] which is provided for Windows and Linux host machines.

The cross-platform framework facilitates a unified testing approach for all team members, enabling organizations to standardize on a test workflow that is independent of the target platform being used or what branch of software is being changed.

== STRIDE software builds are both functional and testable ==
STRIDE enables software builds to be both fully ''functional'' and ''testable'' at the same time. The software works exactly the same as before. Whatever the software image was used for in the past -- system testing, developer debugging, etc. -- is still applicable. The STRIDE [[Test_Units | '''test logic''']] is separated from the application source code and is NOT executed unless invoked via the [[Stride_Runner | '''runner''']]. The [[Source_Instrumentation_Overview | '''source instrumentation''']] is only active when executing tests. The impact of built-in testability to the software application is nominal. The application can easily be switched back to a ''non-testable'' build by simply removing the [[Runtime_Integration#STRIDE_Feature_Control | '''STRIDE_ENABLED''']] preprocessor directive. This automatically controls all STRIDE related source code and macros. There are no changes required to the build process to enable or disable this functionality.

The built-in ''testability'' is similar in concept to a ''debug build'' except the focus is on testing. The ''testable build'' leverages the existing software build process by integrating into the same ''make system'' used by the development team (no one-off or special builds). Automatically included in the build is test automation controllable from the host. When tests are executed, timing analysis is also included with the generated [[Reporting_Model | '''test report''']].
Developers can easily pre-flight test their source code changes before ''committing'' them to a baseline. Builds can be [[Setting_up_your_CI_Environment | '''automatically regression tested''']] as part of the daily build process.

== STRIDE facilitates deeper API/Unit Testing ==
STRIDE offers unique techniques that can be leveraged for deeper API/Unit test coverage. STRIDE provides support for this type of testing [[Test_Units_Overview | '''in C/C++''']]. There are no special APIs required to register tests, suites, etc. Just write your test in any combination of C/C++ and the auto-generated [[Intercept_Module | '''intercept module''']] via the [[Build_Tools | '''STRIDE build tools''']] takes care of everything. Tests from separate teams are automatically aggregated by the system -- no coordination is required.

Writing API/Unit tests in native code is the simplest way to begin validating the software. There is no new language to learn, no proprietary editor, and your normal programming workflow is not interrupted. This type of testing works well for:

* calling APIs directly
* validating C++ classes
* isolating modules
* critical processing / timing
* and much more ...

For more advanced testing scenarios, dependencies can be [[Using_Test_Doubles | '''doubled''']]. This feature provides a means for intercepting C/C++ global functions on the target and substituting a stub, fake, or mock. The substitution is all controllable via the runtime, allowing the software to continue executing normally when not running a test.

[[File_Transfer_Services | ''' File fixturing''']] is another technique that can be leveraged to drive better testing. Test code executing on the target platform can perform file operations on the host remotely. This enables opening, reading, writing, etc. files while executing target-based test logic. File fixturing allows bypassing external input into the system for more controlled and isolated testing.

There are numerous other features that can be leveraged to facilitate deeper API/Unit test coverage:
* [[Test_Macros | '''Assertion macros''']]
* Leveraging [[Test Point Testing in C/C++ | '''Test Points''']] for behavior testing
* Passing parameters to Test Units
* Seamless publishing to [[STRIDE_Test_Space | '''Test Space''']]
* and much [[Test_API | ''' more ... ''']]

== STRIDE includes behavior-based testing techniques ==
STRIDE leverages [[Source_Instrumentation_Overview | '''source instrumentation''']] to provide '' '''behavior-based testing techniques''' '' that can be applied to the executing software application. The execution sequencing of the code, along with data, can be automatically validated based on [[Expectations | '''expectations''']]. Behavior-based testing is different than unit testing or API testing. Behavior testing does not focus on calling functions / methods and validating their return values. Behavior testing validates the ''expected sequencing of the software'' executing under normal operating conditions. This can span threads and process boundaries, and even multiple targets, as the application(s) is running. Leveraging macros -- called [[Test_Point | '''Test Points''']] -- domain experts strategically instrument the source under test. Data can also optionally be associated with a ''' ''Test Point'' ''', which is often critical to validating that the software is behaving as expected. This type of validation can be applied to a wide-range of testing scenerios:

* State Machines
* Data flow through system components
* Sequencing between threads
* Drivers that don't return values
* and much more ..

Another key element to STRIDE based ''behavior testing'' is that the expected code sequencing is automatically validated. This automatic validation can be used for on-going regression testing. The expectations are expressed as simple table definitions along with optional hooks for custom data validation. When failures do occur, context is provided with the file name and associated line number of the unexpected '''Test Point(s)'''. The test validation can be implemented in both [[Expectation_Tests_in_C/C%2B%2B | '''native target code''']] and [[Perl_Script_APIs#STRIDE::Test | '''scripts on the host''']]. In either case, the validation is done without impacting the application's performance (the on-target test code is executed in a background thread and scripts are executed on the host machine).

Software Quality Assurance(SQA) can also add behavior-based testing as part of their existing functional / system testing. Because the [[Stride_Runner | '''runner''']] is a command line utility, it is easily controlled from existing test infrastructure. SQA can leverage the '' '''instrumentation''' '' to create their own [[Reporting_Model#Suites | '''test suites''']] using scripting that executes in concert with existing test automation.

== STRIDE optimizes failure resolution ==
When executing '' '''behavior-based tests''' '' or '' '''API/unit tests''' '', results can be automatically uploaded to [[STRIDE_Test_Space | '''STRIDE Test Space''']] for persistence and analysis. Test result data is uploaded manually (using the web interface) or automatically using the [[Stride_Runner | '''runner''']]. Hosting test results in a ''central location'' with easy access via a browser enables the entire team to better participate in ensuring quality during the ongoing development process. [[STRIDE_Test_Space#Results_View | '''Reports''']] containing test results, timing, [[Tracing | '''tracing logs''']], and built-in test documentation (supported in both [[Test_API#Test_Documentation | '''target code''']] and [[Perl_Script_APIs#Documentation | '''script''']]) all correlated together enhances the context for all team members to manage testing and optimize resolving failures. Team collaboration on testing activities is also facilitated with auto-generated email [[Notifications | '''notifications''']] and [[STRIDE_Test_Space#Messages | '''messaging''']]. Emails contain links to test failures where file names and line numbers are provided to aid in resolution. Messaging allows team members to more effectively communicate on the specifics of test results while minimizing information required in traditional emails, status reports, etc.

[[Category: Overview]]

What is Unique About STRIDE

2010-06-11T23:33:54Z

Mikee: /* STRIDE includes behavior-based testing techniques */

== STRIDE is a cross-platform framework ==
The [[Runtime_Reference | '''STRIDE Runtime''']] is written in standard C on top of a simple [[Platform_Abstraction_Layer | '''platform abstraction layer''']] that enables it to work on virtually any target platform. It is delivered as source code to be included in the application's build system. STRIDE also auto-generates [[Intercept_Module | '''harnessing and remoting logic''']] as source code during the make process, removing any dependencies on specific compilers and / or processors. The transport between the host and target is configurable and supports serial/USB and TCP/IP by default. Custom transports are also available. STRIDE is easily configured for single process or multiple process environments (i.e. Linux, Windows CE, etc.). Testing can also be conducted using an [[Off-Target_Environment | '''off-target environment''']] which is provided for Windows and Linux host machines.

The cross-platform framework facilitates a unified testing approach for all team members, enabling organizations to standardize on a test workflow that is independent of the target platform being used or what branch of software is being changed.

== STRIDE software builds are both functional and testable ==
STRIDE enables software builds to be both fully ''functional'' and ''testable'' at the same time. The software works exactly the same as before. Whatever the software image was used for in the past -- system testing, developer debugging, etc. -- is still applicable. The STRIDE [[Test_Units | '''test logic''']] is separated from the application source code and is NOT executed unless invoked via the [[Stride_Runner | '''runner''']]. The [[Source_Instrumentation_Overview | '''source instrumentation''']] is only active when executing tests. The impact of built-in testability to the software application is nominal. The application can easily be switched back to a ''non-testable'' build by simply removing the [[Runtime_Integration#STRIDE_Feature_Control | '''STRIDE_ENABLED''']] preprocessor directive. This automatically controls all STRIDE related source code and macros. There are no changes required to the build process to enable or disable this functionality.

The built-in ''testability'' is similar in concept to a ''debug build'' except the focus is on testing. The ''testable build'' leverages the existing software build process by integrating into the same ''make system'' used by the development team (no one-off or special builds). Automatically included in the build is test automation controllable from the host. When tests are executed, timing analysis is also included with the generated [[Reporting_Model | '''test report''']].
Developers can easily pre-flight test their source code changes before ''committing'' them to a baseline. Builds can be [[Setting_up_your_CI_Environment | '''automatically regression tested''']] as part of the daily build process.

== STRIDE facilitates deeper API/Unit Testing ==
STRIDE offers unique techniques that can be leveraged for deeper API/Unit test coverage. STRIDE provides support for this type of testing [[Test_Units_Overview | '''in C/C++''']]. There are no special APIs required to register tests, suites, etc. Just write your test in any combination of C/C++ and the auto-generated [[Intercept_Module | '''intercept module''']] via the [[Build_Tools | '''STRIDE build tools''']] takes care of everything. Tests from separate teams are automatically aggregated by the system -- no coordination is required.

Writing API/Unit tests in native code is the simplest way to begin validating the software. There is no new language to learn, no proprietary editor, and your normal programming workflow is not interrupted. This type of testing works well for:

* calling APIs directly
* validating C++ classes
* isolating modules
* critical processing / timing
* and much more ...

For more advanced testing scenarios, dependencies can be [[Using_Test_Doubles | '''doubled''']]. This feature provides a means for intercepting C/C++ global functions on the target and substituting a stub, fake, or mock. The substitution is all controllable via the runtime, allowing the software to continue executing normally when not running a test.

[[File_Transfer_Services | ''' File fixturing''']] is another technique that can be leveraged to drive better testing. Test code executing on the target platform can perform file operations on the host remotely. This enables opening, reading, writing, etc. files while executing target-based test logic. File fixturing allows bypassing external input into the system for more controlled and isolated testing.

There are numerous other features that can be leveraged to facilitate deeper API/Unit test coverage:
* [[Test_Macros | '''Assertion macros''']]
* Leveraging [[Test Point Testing in C/C++ | '''Test Points''']] for behavior testing
* Passing parameters to Test Units
* Seamless publishing to [[STRIDE_Test_Space | '''Test Space''']]
* and much [[Test_API | ''' more ... ''']]

== STRIDE includes behavior-based testing techniques ==
STRIDE leverages [[Source_Instrumentation_Overview | '''source instrumentation''']] to provide '' '''behavior-based testing techniques''' '' that can be applied to the executing software application. The execution sequencing of the code, along with data, can be automatically validated based on [[Expectations | '''expectations''']]. Behavior-based testing is different than unit testing or API testing. Behavior testing does not focus on calling functions / methods and validating their return values. Behavior testing validates the ''expected sequencing of the software'' executing under normal operating conditions. This can span threads and process boundaries, and even multiple targets, as the application(s) is running. Leveraging macros -- called [[Test_Point | '''Test Points''']] -- domain experts strategically instrument the source under test. Data can also optionally be associated with a ''' ''Test Point'' ''', which is often critical to validating that the software is behaving as expected. This type of validation can be applied to a wide-range of testing scenerios:

* State Machines
* Data flow through system components
* Sequencing between threads
* Drivers that don't return values
* and much more ..

Another key element to STRIDE based ''behavior testing'' is that the expected code sequencing is automatically validated. This automatic validation can be used for on-going regression testing. The expectations are expressed as simple table definitions along with optional hooks for custom data validation. When failures do occur, context is provided with the file name and associated line number of the unexpected '''Test Point(s)'''. The test validation can be implemented in both [[Expectation_Tests_in_C/C%2B%2B | '''native target code''']] and [[Perl_Script_APIs#STRIDE::Test | '''scripts on the host''']]. In either case, the validation is done without impacting the application's performance (the on-target test code is executed in a background thread and scripts are executed on the host machine).

Software Quality Assurance(SQA) can also add behavior-based testing as part of their existing functional / system testing. Because the [[Stride_Runner | '''runner''']] is a command line utility, it is easily controlled from existing test infrastructure. SQA can leverage the '' '''instrumentation''' '' to create their own [[Reporting_Model#Suites | '''test suites''']] using scripting that executes in concert with existing test automation.

== STRIDE optimizes failure resolution ==
When executing '' '''behavior-based tests''' '' or '' '''API/unit tests''' '', results can be automatically uploaded to [[STRIDE_Test_Space | '''STRIDE Test Space''']] for storing and analyzing. Result data is uploaded manually (using the web interface) or automatically using the [[Stride_Runner | '''runner''']]. Hosting test results in a ''central location'' with easy access via a browser enables the entire team to better participate in ensuring quality as apart of the development process. [[STRIDE_Test_Space#Results_View | '''Reports''']] containing test results, timing, [[Tracing | '''tracing logs''']], and built-in test documentation (supported in both [[Test_API#Test_Documentation | '''target code''']] and [[Perl_Script_APIs#Documentation | '''script''']]) all correlated together enhances the context for all team members to manage testing and optimize resolving failures. Team collaboration on testing activities is also facilitated with auto-generated email [[Notifications | '''notifications''']] and [[STRIDE_Test_Space#Messages | '''messaging''']]. Emails contain links to test failures where file names and line numbers are provided to aid in resolution. Messaging allows team members to more effectively communicate on the specifics of test results while minimizing information required in traditional emails, status reports, etc.

[[Category: Overview]]

What is Unique About STRIDE

2010-06-11T23:14:32Z

Mikee: /* STRIDE facilitates deeper API/Unit Testing */

== STRIDE is a cross-platform framework ==
The [[Runtime_Reference | '''STRIDE Runtime''']] is written in standard C on top of a simple [[Platform_Abstraction_Layer | '''platform abstraction layer''']] that enables it to work on virtually any target platform. It is delivered as source code to be included in the application's build system. STRIDE also auto-generates [[Intercept_Module | '''harnessing and remoting logic''']] as source code during the make process, removing any dependencies on specific compilers and / or processors. The transport between the host and target is configurable and supports serial/USB and TCP/IP by default. Custom transports are also available. STRIDE is easily configured for single process or multiple process environments (i.e. Linux, Windows CE, etc.). Testing can also be conducted using an [[Off-Target_Environment | '''off-target environment''']] which is provided for Windows and Linux host machines.

The cross-platform framework facilitates a unified testing approach for all team members, enabling organizations to standardize on a test workflow that is independent of the target platform being used or what branch of software is being changed.

== STRIDE software builds are both functional and testable ==
STRIDE enables software builds to be both fully ''functional'' and ''testable'' at the same time. The software works exactly the same as before. Whatever the software image was used for in the past -- system testing, developer debugging, etc. -- is still applicable. The STRIDE [[Test_Units | '''test logic''']] is separated from the application source code and is NOT executed unless invoked via the [[Stride_Runner | '''runner''']]. The [[Source_Instrumentation_Overview | '''source instrumentation''']] is only active when executing tests. The impact of built-in testability to the software application is nominal. The application can easily be switched back to a ''non-testable'' build by simply removing the [[Runtime_Integration#STRIDE_Feature_Control | '''STRIDE_ENABLED''']] preprocessor directive. This automatically controls all STRIDE related source code and macros. There are no changes required to the build process to enable or disable this functionality.

The built-in ''testability'' is similar in concept to a ''debug build'' except the focus is on testing. The ''testable build'' leverages the existing software build process by integrating into the same ''make system'' used by the development team (no one-off or special builds). Automatically included in the build is test automation controllable from the host. When tests are executed, timing analysis is also included with the generated [[Reporting_Model | '''test report''']].
Developers can easily pre-flight test their source code changes before ''committing'' them to a baseline. Builds can be [[Setting_up_your_CI_Environment | '''automatically regression tested''']] as part of the daily build process.

== STRIDE facilitates deeper API/Unit Testing ==
STRIDE offers unique techniques that can be leveraged for deeper API/Unit test coverage. STRIDE provides support for this type of testing [[Test_Units_Overview | '''in C/C++''']]. There are no special APIs required to register tests, suites, etc. Just write your test in any combination of C/C++ and the auto-generated [[Intercept_Module | '''intercept module''']] via the [[Build_Tools | '''STRIDE build tools''']] takes care of everything. Tests from separate teams are automatically aggregated by the system -- no coordination is required.

Writing API/Unit tests in native code is the simplest way to begin validating the software. There is no new language to learn, no proprietary editor, and your normal programming workflow is not interrupted. This type of testing works well for:

* calling APIs directly
* validating C++ classes
* isolating modules
* critical processing / timing
* and much more ...

For more advanced testing scenarios, dependencies can be [[Using_Test_Doubles | '''doubled''']]. This feature provides a means for intercepting C/C++ global functions on the target and substituting a stub, fake, or mock. The substitution is all controllable via the runtime, allowing the software to continue executing normally when not running a test.

[[File_Transfer_Services | ''' File fixturing''']] is another technique that can be leveraged to drive better testing. Test code executing on the target platform can perform file operations on the host remotely. This enables opening, reading, writing, etc. files while executing target-based test logic. File fixturing allows bypassing external input into the system for more controlled and isolated testing.

There are numerous other features that can be leveraged to facilitate deeper API/Unit test coverage:
* [[Test_Macros | '''Assertion macros''']]
* Leveraging [[Test Point Testing in C/C++ | '''Test Points''']] for behavior testing
* Passing parameters to Test Units
* Seamless publishing to [[STRIDE_Test_Space | '''Test Space''']]
* and much [[Test_API | ''' more ... ''']]

== STRIDE includes behavior-based testing techniques ==
STRIDE leverages [[Source_Instrumentation_Overview | '''source instrumentation''']] to provide '' '''behavior-based testing techniques''' '' that can be applied to the executing software application. The execution sequencing of the code, along with data, can be automatically validated based on [[Expectations | '''expectations''']]. Behavior-based testing is different than unit testing or API testing. Behavior testing does not focus on calling functions / methods and validating their return values. Behavior testing validates the ''expected sequencing of the software'' executing under normal operating conditions. This can span threads and process boundaries, and even multiple targets, as the application(s) is running. Leveraging macros -- called [[Test_Point | '''Test Points''']] -- domain experts strategically instrument the source under test. Data can also optionally be associated with a ''' ''Test Point'' ''', which is often critical to validating that the software is behaving as expected. This type of validation can be applied to a wide-range of testing scenerios:

* State Machines
* Data flow through system components
* Sequencing between threads
* Drivers that don't return values
* and much more ..

Another key element to STRIDE based ''behavior testing'' is that the expected code sequencing is automatically validated. This automatic validation can be used for on-going regression testing. The expectations are realized by simple table definitions along with hooks for customization related to optional data. When failures do occur, context is provided with the file name and associated line number of the unexpected '''Test Point(s)''' behavior. The test validation can be implemented in both [[Expectation_Tests_in_C/C%2B%2B | '''native target code''']] and [[Perl_Script_APIs#STRIDE::Test | '''scripting on the host''']]. In either case, the validation is done without impacting the application's performance (i.e. target code executed in a background thread and host script executed on desktop).

Software Quality Assurance(SQA) can also add behavior-based testing as part of their existing functional / system testing. Because the [[Stride_Runner | '''runner''']] is a command line utility, it is easily controlled from existing test infrastructure. SQA can also leverage the '' '''instrumentation''' '' to create their own [[Reporting_Model#Suites | '''test suites''']] using scripting that executes in concert with existing test automation.

== STRIDE optimizes failure resolution ==
When executing '' '''behavior-based tests''' '' or '' '''API/unit tests''' '', results can be automatically uploaded to [[STRIDE_Test_Space | '''STRIDE Test Space''']] for storing and analyzing. Result data is uploaded manually (using the web interface) or automatically using the [[Stride_Runner | '''runner''']]. Hosting test results in a ''central location'' with easy access via a browser enables the entire team to better participate in ensuring quality as apart of the development process. [[STRIDE_Test_Space#Results_View | '''Reports''']] containing test results, timing, [[Tracing | '''tracing logs''']], and built-in test documentation (supported in both [[Test_API#Test_Documentation | '''target code''']] and [[Perl_Script_APIs#Documentation | '''script''']]) all correlated together enhances the context for all team members to manage testing and optimize resolving failures. Team collaboration on testing activities is also facilitated with auto-generated email [[Notifications | '''notifications''']] and [[STRIDE_Test_Space#Messages | '''messaging''']]. Emails contain links to test failures where file names and line numbers are provided to aid in resolution. Messaging allows team members to more effectively communicate on the specifics of test results while minimizing information required in traditional emails, status reports, etc.

[[Category: Overview]]

What is Unique About STRIDE

2010-06-11T23:05:27Z

Mikee: /* STRIDE facilitates deeper API/Unit Testing */

== STRIDE is a cross-platform framework ==
The [[Runtime_Reference | '''STRIDE Runtime''']] is written in standard C on top of a simple [[Platform_Abstraction_Layer | '''platform abstraction layer''']] that enables it to work on virtually any target platform. It is delivered as source code to be included in the application's build system. STRIDE also auto-generates [[Intercept_Module | '''harnessing and remoting logic''']] as source code during the make process, removing any dependencies on specific compilers and / or processors. The transport between the host and target is configurable and supports serial/USB and TCP/IP by default. Custom transports are also available. STRIDE is easily configured for single process or multiple process environments (i.e. Linux, Windows CE, etc.). Testing can also be conducted using an [[Off-Target_Environment | '''off-target environment''']] which is provided for Windows and Linux host machines.

The cross-platform framework facilitates a unified testing approach for all team members, enabling organizations to standardize on a test workflow that is independent of the target platform being used or what branch of software is being changed.

== STRIDE software builds are both functional and testable ==
STRIDE enables software builds to be both fully ''functional'' and ''testable'' at the same time. The software works exactly the same as before. Whatever the software image was used for in the past -- system testing, developer debugging, etc. -- is still applicable. The STRIDE [[Test_Units | '''test logic''']] is separated from the application source code and is NOT executed unless invoked via the [[Stride_Runner | '''runner''']]. The [[Source_Instrumentation_Overview | '''source instrumentation''']] is only active when executing tests. The impact of built-in testability to the software application is nominal. The application can easily be switched back to a ''non-testable'' build by simply removing the [[Runtime_Integration#STRIDE_Feature_Control | '''STRIDE_ENABLED''']] preprocessor directive. This automatically controls all STRIDE related source code and macros. There are no changes required to the build process to enable or disable this functionality.

The built-in ''testability'' is similar in concept to a ''debug build'' except the focus is on testing. The ''testable build'' leverages the existing software build process by integrating into the same ''make system'' used by the development team (no one-off or special builds). Automatically included in the build is test automation controllable from the host. When tests are executed, timing analysis is also included with the generated [[Reporting_Model | '''test report''']].
Developers can easily pre-flight test their source code changes before ''committing'' them to a baseline. Builds can be [[Setting_up_your_CI_Environment | '''automatically regression tested''']] as part of the daily build process.

== STRIDE facilitates deeper API/Unit Testing ==
STRIDE offers unique techniques that can be leveraged for deeper API/Unit test coverage. STRIDE provides support for this type of testing [[Test_Units_Overview | '''in C/C++''']]. There are no special APIs required to register tests, suites, etc. Just write your test in any combination of C/C++ and the auto-generated [[Intercept_Module | '''intercept module''']] via the [[Build_Tools | '''STRIDE build tools''']] takes care of everything. Tests from separate teams are automatically aggregated by the system -- no coordination is required.

Writing API/Unit tests in native code is the simplest way to begin validating the software. There is no new language to learn, no proprietary editor, and your normal programming workflow is not interrupted. This type of testing works well for:

* calling APIs directly
* validating C++ classes
* isolating modules
* critical processing / timing
* and much more ...

For more advanced testing scenarios, dependencies can be [[Using_Test_Doubles | '''doubled''']]. This feature provides a means for intercepting C/C++ global functions on the target and substituting a stub, fake, or mock. The substitution is all controllable via the [[Stride_Runner | '''runner''']], allowing the software to continue executing normally when not running a test.

[[File_Transfer_Services | ''' File fixturing''']] is another technique that can be leveraged to drive better testing. Test code executing on the target platform can perform file operations on the host remotely. This enables opening, reading, writing, etc. files while executing target-based test logic. File fixturing allows bypassing external input into the system for more controlled and isolated testing.

There are numerous other features that can be leveraged to facilitate deeper API/Unit test coverage:
* [[Test_Macros | '''Assertion macros''']]
* Leveraging [[Test Point Testing in C/C++ | '''Test Points''']] for behavior testing
* Passing parameters to Test Units
* Seamless publishing to [[STRIDE_Test_Space | '''Test Space''']]
* and much [[Test_API | ''' more ... ''']]

== STRIDE includes behavior-based testing techniques ==
STRIDE leverages [[Source_Instrumentation_Overview | '''source instrumentation''']] to provide '' '''behavior-based testing techniques''' '' that can be applied to the executing software application. The execution sequencing of the code, along with data, can be automatically validated based on [[Expectations | '''expectations''']]. Behavior-based testing is different than unit testing or API testing. Behavior testing does not focus on calling functions / methods and validating their return values. Behavior testing validates the ''expected sequencing of the software'' executing under normal operating conditions. This can span threads and process boundaries, and even multiple targets, as the application(s) is running. Leveraging macros -- called [[Test_Point | '''Test Points''']] -- domain experts strategically instrument the source under test. Data can also optionally be associated with a ''' ''Test Point'' ''', which is often critical to validating that the software is behaving as expected. This type of validation can be applied to a wide-range of testing scenerios:

* State Machines
* Data flow through system components
* Sequencing between threads
* Drivers that don't return values
* and much more ..

Another key element to STRIDE based ''behavior testing'' is that the expected code sequencing is automatically validated. This automatic validation can be used for on-going regression testing. The expectations are realized by simple table definitions along with hooks for customization related to optional data. When failures do occur, context is provided with the file name and associated line number of the unexpected '''Test Point(s)''' behavior. The test validation can be implemented in both [[Expectation_Tests_in_C/C%2B%2B | '''native target code''']] and [[Perl_Script_APIs#STRIDE::Test | '''scripting on the host''']]. In either case, the validation is done without impacting the application's performance (i.e. target code executed in a background thread and host script executed on desktop).

Software Quality Assurance(SQA) can also add behavior-based testing as part of their existing functional / system testing. Because the [[Stride_Runner | '''runner''']] is a command line utility, it is easily controlled from existing test infrastructure. SQA can also leverage the '' '''instrumentation''' '' to create their own [[Reporting_Model#Suites | '''test suites''']] using scripting that executes in concert with existing test automation.

== STRIDE optimizes failure resolution ==
When executing '' '''behavior-based tests''' '' or '' '''API/unit tests''' '', results can be automatically uploaded to [[STRIDE_Test_Space | '''STRIDE Test Space''']] for storing and analyzing. Result data is uploaded manually (using the web interface) or automatically using the [[Stride_Runner | '''runner''']]. Hosting test results in a ''central location'' with easy access via a browser enables the entire team to better participate in ensuring quality as apart of the development process. [[STRIDE_Test_Space#Results_View | '''Reports''']] containing test results, timing, [[Tracing | '''tracing logs''']], and built-in test documentation (supported in both [[Test_API#Test_Documentation | '''target code''']] and [[Perl_Script_APIs#Documentation | '''script''']]) all correlated together enhances the context for all team members to manage testing and optimize resolving failures. Team collaboration on testing activities is also facilitated with auto-generated email [[Notifications | '''notifications''']] and [[STRIDE_Test_Space#Messages | '''messaging''']]. Emails contain links to test failures where file names and line numbers are provided to aid in resolution. Messaging allows team members to more effectively communicate on the specifics of test results while minimizing information required in traditional emails, status reports, etc.

[[Category: Overview]]

What is Unique About STRIDE

2010-06-11T23:01:03Z

Mikee: /* STRIDE software builds are both functional and testable */

== STRIDE is a cross-platform framework ==
The [[Runtime_Reference | '''STRIDE Runtime''']] is written in standard C on top of a simple [[Platform_Abstraction_Layer | '''platform abstraction layer''']] that enables it to work on virtually any target platform. It is delivered as source code to be included in the application's build system. STRIDE also auto-generates [[Intercept_Module | '''harnessing and remoting logic''']] as source code during the make process, removing any dependencies on specific compilers and / or processors. The transport between the host and target is configurable and supports serial/USB and TCP/IP by default. Custom transports are also available. STRIDE is easily configured for single process or multiple process environments (i.e. Linux, Windows CE, etc.). Testing can also be conducted using an [[Off-Target_Environment | '''off-target environment''']] which is provided for Windows and Linux host machines.

The cross-platform framework facilitates a unified testing approach for all team members, enabling organizations to standardize on a test workflow that is independent of the target platform being used or what branch of software is being changed.

== STRIDE software builds are both functional and testable ==
STRIDE enables software builds to be both fully ''functional'' and ''testable'' at the same time. The software works exactly the same as before. Whatever the software image was used for in the past -- system testing, developer debugging, etc. -- is still applicable. The STRIDE [[Test_Units | '''test logic''']] is separated from the application source code and is NOT executed unless invoked via the [[Stride_Runner | '''runner''']]. The [[Source_Instrumentation_Overview | '''source instrumentation''']] is only active when executing tests. The impact of built-in testability to the software application is nominal. The application can easily be switched back to a ''non-testable'' build by simply removing the [[Runtime_Integration#STRIDE_Feature_Control | '''STRIDE_ENABLED''']] preprocessor directive. This automatically controls all STRIDE related source code and macros. There are no changes required to the build process to enable or disable this functionality.

The built-in ''testability'' is similar in concept to a ''debug build'' except the focus is on testing. The ''testable build'' leverages the existing software build process by integrating into the same ''make system'' used by the development team (no one-off or special builds). Automatically included in the build is test automation controllable from the host. When tests are executed, timing analysis is also included with the generated [[Reporting_Model | '''test report''']].
Developers can easily pre-flight test their source code changes before ''committing'' them to a baseline. Builds can be [[Setting_up_your_CI_Environment | '''automatically regression tested''']] as part of the daily build process.

== STRIDE facilitates deeper API/Unit Testing ==
STRIDE offers unique techniques that can be leveraged for deeper API/Unit test coverage. STRIDE provides support for this type of testing [[Test_Units_Overview | '''in C/C++''']]. There are no special APIs required to register tests, suites, etc. Just write your test in any combination of C/C++ and the auto-generated [[Intercept_Module | '''intercept module''']] via the [[Build_Tools | '''STRIDE build tools''']] takes care of everything. Also aggregating tests from separate teams is supported -- no coordination is required. Writing API/Unit tests in native code is the simplest way to begin validating the software. There is no new language to learn, no proprietary editor, and the same workflow used when programming can be leveraged. This type of testing works well for:

* calling APIs directly
* validating C++ classes
* isolating modules
* critical processing / timing
* and much more ...

For more advanced testing scenarios, dependencies can be [[Using_Test_Doubles | '''doubled''']]. This feature provides a means for intercepting C/C++ language global functions on the target and substitute a stub, fake, or mock. The substitution is all controllable via the [[Stride_Runner | '''runner''']], allowing the software to continue executing normally when not running a test.

[[File_Transfer_Services | ''' File fixturing''']] is another technique that can be leveraged to drive better testing. Test code executing on the target platform can perform file operations on the host remotely. This enables opening, reading, writing, etc. files while executing target-based test logic. File fixturing allows bypassing external input into the system for more controlled and isolated testing.

There are numerous other features that can be leveraged to facilitate deeper API/Unit test coverage:
* [[Test_Macros | '''Assertion macros''']]
* Leveraging [[Test Point Testing in C/C++ | '''Test Points''']] for behavior testing
* Passing parameters to Test Units
* Seamless publishing to [[STRIDE_Test_Space | '''Test Space''']]
* and much [[Test_API | ''' more ... ''']]

== STRIDE includes behavior-based testing techniques ==
STRIDE leverages [[Source_Instrumentation_Overview | '''source instrumentation''']] to provide '' '''behavior-based testing techniques''' '' that can be applied to the executing software application. The execution sequencing of the code, along with data, can be automatically validated based on [[Expectations | '''expectations''']]. Behavior-based testing is different than unit testing or API testing. Behavior testing does not focus on calling functions / methods and validating their return values. Behavior testing validates the ''expected sequencing of the software'' executing under normal operating conditions. This can span threads and process boundaries, and even multiple targets, as the application(s) is running. Leveraging macros -- called [[Test_Point | '''Test Points''']] -- domain experts strategically instrument the source under test. Data can also optionally be associated with a ''' ''Test Point'' ''', which is often critical to validating that the software is behaving as expected. This type of validation can be applied to a wide-range of testing scenerios:

* State Machines
* Data flow through system components
* Sequencing between threads
* Drivers that don't return values
* and much more ..

Another key element to STRIDE based ''behavior testing'' is that the expected code sequencing is automatically validated. This automatic validation can be used for on-going regression testing. The expectations are realized by simple table definitions along with hooks for customization related to optional data. When failures do occur, context is provided with the file name and associated line number of the unexpected '''Test Point(s)''' behavior. The test validation can be implemented in both [[Expectation_Tests_in_C/C%2B%2B | '''native target code''']] and [[Perl_Script_APIs#STRIDE::Test | '''scripting on the host''']]. In either case, the validation is done without impacting the application's performance (i.e. target code executed in a background thread and host script executed on desktop).

Software Quality Assurance(SQA) can also add behavior-based testing as part of their existing functional / system testing. Because the [[Stride_Runner | '''runner''']] is a command line utility, it is easily controlled from existing test infrastructure. SQA can also leverage the '' '''instrumentation''' '' to create their own [[Reporting_Model#Suites | '''test suites''']] using scripting that executes in concert with existing test automation.

== STRIDE optimizes failure resolution ==
When executing '' '''behavior-based tests''' '' or '' '''API/unit tests''' '', results can be automatically uploaded to [[STRIDE_Test_Space | '''STRIDE Test Space''']] for storing and analyzing. Result data is uploaded manually (using the web interface) or automatically using the [[Stride_Runner | '''runner''']]. Hosting test results in a ''central location'' with easy access via a browser enables the entire team to better participate in ensuring quality as apart of the development process. [[STRIDE_Test_Space#Results_View | '''Reports''']] containing test results, timing, [[Tracing | '''tracing logs''']], and built-in test documentation (supported in both [[Test_API#Test_Documentation | '''target code''']] and [[Perl_Script_APIs#Documentation | '''script''']]) all correlated together enhances the context for all team members to manage testing and optimize resolving failures. Team collaboration on testing activities is also facilitated with auto-generated email [[Notifications | '''notifications''']] and [[STRIDE_Test_Space#Messages | '''messaging''']]. Emails contain links to test failures where file names and line numbers are provided to aid in resolution. Messaging allows team members to more effectively communicate on the specifics of test results while minimizing information required in traditional emails, status reports, etc.

[[Category: Overview]]

What is Unique About STRIDE

2010-06-11T22:58:46Z

Mikee: /* STRIDE is a cross-platform framework */

== STRIDE is a cross-platform framework ==
The [[Runtime_Reference | '''STRIDE Runtime''']] is written in standard C on top of a simple [[Platform_Abstraction_Layer | '''platform abstraction layer''']] that enables it to work on virtually any target platform. It is delivered as source code to be included in the application's build system. STRIDE also auto-generates [[Intercept_Module | '''harnessing and remoting logic''']] as source code during the make process, removing any dependencies on specific compilers and / or processors. The transport between the host and target is configurable and supports serial/USB and TCP/IP by default. Custom transports are also available. STRIDE is easily configured for single process or multiple process environments (i.e. Linux, Windows CE, etc.). Testing can also be conducted using an [[Off-Target_Environment | '''off-target environment''']] which is provided for Windows and Linux host machines.

The cross-platform framework facilitates a unified testing approach for all team members, enabling organizations to standardize on a test workflow that is independent of the target platform being used or what branch of software is being changed.

== STRIDE software builds are both functional and testable ==
STRIDE enables software builds to be both fully ''functional'' and ''testable'' at the same time. The software works exactly the same as before. Whatever the software image was used for in the past -- system testing, developer debugging, etc. -- is still applicable. The STRIDE [[Test_Units | '''test logic''']] is separated from the application source code and is NOT executed unless invoked via the [[Stride_Runner | '''runner''']]. The [[Source_Instrumentation_Overview | '''source instrumentation''']] is only active when executing tests. The impact of built-in testability to the software application is nominal. The application can easily be switched back to a ''non-testable'' build by simply removing the [[Runtime_Integration#STRIDE_Feature_Control | '''STRIDE_ENABLED''']] preprocessor directive. This automatically controls all STRIDE related source code and macros. There are no changes required to the build process to enable or disable this functionality.

The built-in ''testability'' is similar in concept to a ''debug build'' except the focus is on testing. The ''testable build'' leverages the existing software build process by integrating into the same ''make system'' used by the development team (no one-off or special builds). Automatically included in the build is test automation controllable from the host. When tests are executed timing analysis is also included with the generated [[Reporting_Model | '''test report''']].
Developers can pre-flight test their source code changes before ''committing'' them to a baseline. Builds can be [[Setting_up_your_CI_Environment | '''automatically regression''']] tested as part of the daily build process.

== STRIDE facilitates deeper API/Unit Testing ==
STRIDE offers unique techniques that can be leveraged for deeper API/Unit test coverage. STRIDE provides support for this type of testing [[Test_Units_Overview | '''in C/C++''']]. There are no special APIs required to register tests, suites, etc. Just write your test in any combination of C/C++ and the auto-generated [[Intercept_Module | '''intercept module''']] via the [[Build_Tools | '''STRIDE build tools''']] takes care of everything. Also aggregating tests from separate teams is supported -- no coordination is required. Writing API/Unit tests in native code is the simplest way to begin validating the software. There is no new language to learn, no proprietary editor, and the same workflow used when programming can be leveraged. This type of testing works well for:

* calling APIs directly
* validating C++ classes
* isolating modules
* critical processing / timing
* and much more ...

For more advanced testing scenarios, dependencies can be [[Using_Test_Doubles | '''doubled''']]. This feature provides a means for intercepting C/C++ language global functions on the target and substitute a stub, fake, or mock. The substitution is all controllable via the [[Stride_Runner | '''runner''']], allowing the software to continue executing normally when not running a test.

[[File_Transfer_Services | ''' File fixturing''']] is another technique that can be leveraged to drive better testing. Test code executing on the target platform can perform file operations on the host remotely. This enables opening, reading, writing, etc. files while executing target-based test logic. File fixturing allows bypassing external input into the system for more controlled and isolated testing.

There are numerous other features that can be leveraged to facilitate deeper API/Unit test coverage:
* [[Test_Macros | '''Assertion macros''']]
* Leveraging [[Test Point Testing in C/C++ | '''Test Points''']] for behavior testing
* Passing parameters to Test Units
* Seamless publishing to [[STRIDE_Test_Space | '''Test Space''']]
* and much [[Test_API | ''' more ... ''']]

== STRIDE includes behavior-based testing techniques ==
STRIDE leverages [[Source_Instrumentation_Overview | '''source instrumentation''']] to provide '' '''behavior-based testing techniques''' '' that can be applied to the executing software application. The execution sequencing of the code, along with data, can be automatically validated based on [[Expectations | '''expectations''']]. Behavior-based testing is different than unit testing or API testing. Behavior testing does not focus on calling functions / methods and validating their return values. Behavior testing validates the ''expected sequencing of the software'' executing under normal operating conditions. This can span threads and process boundaries, and even multiple targets, as the application(s) is running. Leveraging macros -- called [[Test_Point | '''Test Points''']] -- domain experts strategically instrument the source under test. Data can also optionally be associated with a ''' ''Test Point'' ''', which is often critical to validating that the software is behaving as expected. This type of validation can be applied to a wide-range of testing scenerios:

* State Machines
* Data flow through system components
* Sequencing between threads
* Drivers that don't return values
* and much more ..

Another key element to STRIDE based ''behavior testing'' is that the expected code sequencing is automatically validated. This automatic validation can be used for on-going regression testing. The expectations are realized by simple table definitions along with hooks for customization related to optional data. When failures do occur, context is provided with the file name and associated line number of the unexpected '''Test Point(s)''' behavior. The test validation can be implemented in both [[Expectation_Tests_in_C/C%2B%2B | '''native target code''']] and [[Perl_Script_APIs#STRIDE::Test | '''scripting on the host''']]. In either case, the validation is done without impacting the application's performance (i.e. target code executed in a background thread and host script executed on desktop).

Software Quality Assurance(SQA) can also add behavior-based testing as part of their existing functional / system testing. Because the [[Stride_Runner | '''runner''']] is a command line utility, it is easily controlled from existing test infrastructure. SQA can also leverage the '' '''instrumentation''' '' to create their own [[Reporting_Model#Suites | '''test suites''']] using scripting that executes in concert with existing test automation.

== STRIDE optimizes failure resolution ==
When executing '' '''behavior-based tests''' '' or '' '''API/unit tests''' '', results can be automatically uploaded to [[STRIDE_Test_Space | '''STRIDE Test Space''']] for storing and analyzing. Result data is uploaded manually (using the web interface) or automatically using the [[Stride_Runner | '''runner''']]. Hosting test results in a ''central location'' with easy access via a browser enables the entire team to better participate in ensuring quality as apart of the development process. [[STRIDE_Test_Space#Results_View | '''Reports''']] containing test results, timing, [[Tracing | '''tracing logs''']], and built-in test documentation (supported in both [[Test_API#Test_Documentation | '''target code''']] and [[Perl_Script_APIs#Documentation | '''script''']]) all correlated together enhances the context for all team members to manage testing and optimize resolving failures. Team collaboration on testing activities is also facilitated with auto-generated email [[Notifications | '''notifications''']] and [[STRIDE_Test_Space#Messages | '''messaging''']]. Emails contain links to test failures where file names and line numbers are provided to aid in resolution. Messaging allows team members to more effectively communicate on the specifics of test results while minimizing information required in traditional emails, status reports, etc.

[[Category: Overview]]

STRIDE Overview

2010-06-11T22:54:16Z

Mikee:

STRIDE™ has been designed specifically for ''on-target white-box testing''. STRIDE™ is a test system used for validating embedded software executing on-target. The STRIDE™ system consists of a '' '''Framework''' '' for testing and a hosted '' '''Web Application''' '' for storing and analyzing test results.

The '' '''Framework''' '' includes a ''cross-platform'' [[Runtime_Reference | '''runtime''']] source package that supports connectivity with the host system and provides [[Runtime_Test_Services | '''services for testing''']] and [[Source_Instrumentation_Overview | '''source instrumentation''']]. The '' '''runtime''' '' enables testability to be compiled into the embedded software with minimal impact on performance or the size of the application. The '' '''Framework''' '' also contains a host-based [[Stride_Runner | '''runner''']] for interactive and automated test execution and publishing to our [[STRIDE_Test_Space | '''hosted web application''']].

[[image:STRIDE 4.2.jpg | 500px]]

STRIDE's test system is designed to deliver a broad spectrum of ''testing capabilities'' that enable teams to test earlier and more effectively. Developers can implement [[Test_Units_Overview | '''API and unit tests''']] in C/C++ that execute on On-Target and are controlled from the host. Unique features such as [[File_Transfer_Services | '''file fixturing''']] and [[Using_Test_Doubles | '''function doubling''']] facilitate deeper coverage of your software components. Because the '' '''Framework''' '' is ''cross-platform'', testing can also be executed using Off-Target environments such as Windows and Linux host machines. The [[Linux SDK | '''Linux''']] and [[Windows SDK | '''Windows''']] ''SDKs'' allow for a seamless transition between the real target and an off-target host environment.

Beyond traditional developer testing, STRIDE also provides [[Test_Modules_Overview |'''behavior-based testing techniques''']]. Behavior-based testing is different than unit testing or API testing in that it ''does not'' focus on calling functions and validating return values. Behavior testing, rather, validates the ''expected sequencing of the software'' executing under normal operating conditions. This can span threads and process boundaries, and even multiple targets, as the application(s) is running. Developers insert instrumentation [[Test_Point | '''macros''']] to enable this approach, making it especially effective for testing legacy code. This kind of ''expectation-based'' testing can be used to validate broadly scoped scenarios that span system-wide, for example:

* Complete data flow through system components
* Communication between threads, processes, and targets
* Behavior of stacks, state machines, and drivers
* … and much more

Once tests have been implemented, they are executed using the '' '''runner''' '' and test reports can be automatically uploaded to our '''web application''' - [[STRIDE_Test_Space | '''STRIDE Test Space''']]. This application allows all team members to track and collaborate on results. Failure resolution is optimized by centralizing results, providing specific source information related to failures, and automatic email notification for new results.


 


'''For more details refer to the following:'''

* '''[[What is Unique About STRIDE]]'''

* '''[[Types of Testing Supported]]'''

* '''[[Frequently Asked Questions About STRIDE | Frequently Asked Questions]]'''



[[Category: Overview]]

STRIDE Overview

2010-06-11T22:50:35Z

Mikee:

STRIDE™ has been designed specifically for ''on-target white-box testing''. STRIDE™ is a test system used for validating embedded software executing on-target. The STRIDE™ system consists of a '' '''Framework''' '' for testing and a hosted '' '''Web Application''' '' for storing and analyzing test results.

The '' '''Framework''' '' includes a ''cross-platform'' [[Runtime_Reference | '''runtime''']] source package that supports connectivity with the host system and provides [[Runtime_Test_Services | '''services for testing''']] and [[Source_Instrumentation_Overview | '''source instrumentation''']]. The '' '''runtime''' '' enables testability to be compiled into the embedded software with minimal impact on performance or the size of the application. The '' '''Framework''' '' also contains a host-based [[Stride_Runner | '''runner''']] for interactive and automated test execution and publishing to our [[STRIDE_Test_Space | '''hosted web application''']].

[[image:STRIDE 4.2.jpg | 500px]]

STRIDE's test system is designed to deliver a broad spectrum of ''testing capabilities'' that enable teams to test earlier and more effectively. Developers can implement [[Test_Units_Overview | '''API and unit tests''']] in C/C++ that execute on On-Target and are controlled from the host. Unique features such as [[File_Transfer_Services | '''file fixturing''']] and [[Using_Test_Doubles | '''function doubling''']] facilitate deeper coverage of your software components. Because the '' '''Framework''' '' is ''cross-platform'', testing can also be executed using Off-Target environments such as Windows and Linux host machines. The [[Linux SDK | '''Linux''']] and [[Windows SDK | '''Windows''']] ''SDKs'' allow for a seamless transition between the real target and an Off-Target host environment.

Beyond traditional developer testing, STRIDE also provides [[Test_Modules_Overview |'''behavior-based testing techniques''']]. Behavior-based testing is different than unit testing or API testing in that it ''does not'' focus on calling functions and validating return values. Behavior testing, rather, validates the ''expected sequencing of the software'' executing under normal operating conditions. This can span threads and process boundaries, and even multiple targets, as the application(s) is running. Developers insert instrumentation [[Test_Point | '''macros''']] to enable this approach, making it especially effective for testing legacy code. This kind of ''expectation-based'' testing can be used to validate much more comprehensive and system wide testing scenarios, for example:

* Data flow through system components
* Communication between threads, processes, and targets
* Behavior of stacks, state machines, and drivers
* … and much more

Once tests have been implemented, they are executed using the '' '''runner''' '' and test reports can be automatically uploaded to our '''Web application''' - [[STRIDE_Test_Space | '''STRIDE Test Space''']]. This application allows all team members to track and collaborate on results. Failure resolution is optimized by centralizing results, providing specific source information related to failures, and automatic email notification for new results.


 


'''For more details refer to the following:'''

* '''[[What is Unique About STRIDE]]'''

* '''[[Types of Testing Supported]]'''

* '''[[Frequently Asked Questions About STRIDE | Frequently Asked Questions]]'''



[[Category: Overview]]

Types of Testing Supported by STRIDE

2010-06-11T22:40:16Z

Mikee: /* Behavior Testing */

== Overview ==

One of the questions that should be asked is ''what is the '''value''' of the test?'' If the test does not discover any defects or does not provide ongoing regression, then the value is questionable. Also ''what is the '''effort''' in implementing the test?'' STRIDE has been uniquely designed to support maximizing the '' '''value''' '' of the test while minimizing the '' '''effort''' '' to implement it.

The STRIDE test system supports three general types of testing:
* '''Unit Testing'''
* '''API Testing'''
* '''Behavior Testing'''

== Unit Testing ==
[[Test Units Overview | STRIDE Unit Testing]] supports the model found in typical [http://en.wikipedia.org/wiki/XUnit xUnit-style] testing frameworks. STRIDE also offers a number of [[Test API | additional features]] optimized for testing embedded software. Traditional '''Unit Testing''' presents a number of challenges in testing embedded software:

* Testing functions/classes in '''isolation''' requires a lot of extra work, especially if your software was not designed upfront for testability
* Testing legacy software might have limited value, particularly if the software is stable with respect to defects
* The software is often not well suited for '''others''' to participate in the test implementation, since there is too much internal knowledge required to be productive
* It can be difficult to automate execution of the full set of tests on the real target device

== API Testing ==
STRIDE supports '''API Testing''' by leveraging the same [[Test Units Overview | techniques]] available for Unit Testing. API Testing differs from unit testing in that the tests focus on direct testing of a well-defined interface. This kind of testing is typically easy to scope depends on the size of the interface and often has a better ''return-on-effort''. The design of ''public interfaces'' often lends itself to testing in isolation ''without'' implementing special test logic (i.e. no stubbing required), which make the test implementation simpler. What's more, public APIs are most likely documented. As a result, ''non domain experts'' can more easily participate in the test implementation. Although API Testing often represents a smaller percentage of the software being exercised, the '''value''' and '''effort''' required is well understood.

== Behavior Testing ==
STRIDE also supports '''Behavior Testing''', which is different than Unit Testing or API Testing in that it does not focus on calling functions and validating return values. Behavior Testing, rather, validates the ''expected sequencing and state of the software'' executing under normal operating conditions. To learn more about the uniqueness of Behavior Testing [[What_is_Unique_About_STRIDE#STRIDE_includes_behavior-based_testing_techniques | '''read here''']]. We believe that Behavior Testing has a very high ''return-on-effort'' and can be easily deployed into legacy software systems.

Some of the advantages of '''Behavior Testing''' are:
* Tests are performed with fully functional software builds -- there are no code isolation issues challenges
* The technique is easy to grasp and is applicable to a large percentage of code bases
* The instrumentation is non-obtrusive and allows other functional and black-box testing to be executed on the same build
* It's aasy to add [[Test Point | Test Points]] to provide coverage
* Allows other technical resources to participate in test implementation
* Tests can be written using [[Perl_Script_APIs#STRIDE::Test | scripting on the host]] and [[Expectation_Tests_in_C/C%2B%2B | native target code]]
* Validation is fully automated and thus does not rely on manual interpretation by domain experts
* Design knowledge is extracted and made available to other team members via [[Source Instrumentation Overview | instrumentation]]

[[Category: Overview]]

Types of Testing Supported by STRIDE

2010-06-11T22:36:00Z

Mikee: /* Behavior Testing */

== Overview ==

One of the questions that should be asked is ''what is the '''value''' of the test?'' If the test does not discover any defects or does not provide ongoing regression, then the value is questionable. Also ''what is the '''effort''' in implementing the test?'' STRIDE has been uniquely designed to support maximizing the '' '''value''' '' of the test while minimizing the '' '''effort''' '' to implement it.

The STRIDE test system supports three general types of testing:
* '''Unit Testing'''
* '''API Testing'''
* '''Behavior Testing'''

== Unit Testing ==
[[Test Units Overview | STRIDE Unit Testing]] supports the model found in typical [http://en.wikipedia.org/wiki/XUnit xUnit-style] testing frameworks. STRIDE also offers a number of [[Test API | additional features]] optimized for testing embedded software. Traditional '''Unit Testing''' presents a number of challenges in testing embedded software:

* Testing functions/classes in '''isolation''' requires a lot of extra work, especially if your software was not designed upfront for testability
* Testing legacy software might have limited value, particularly if the software is stable with respect to defects
* The software is often not well suited for '''others''' to participate in the test implementation, since there is too much internal knowledge required to be productive
* It can be difficult to automate execution of the full set of tests on the real target device

== API Testing ==
STRIDE supports '''API Testing''' by leveraging the same [[Test Units Overview | techniques]] available for Unit Testing. API Testing differs from unit testing in that the tests focus on direct testing of a well-defined interface. This kind of testing is typically easy to scope depends on the size of the interface and often has a better ''return-on-effort''. The design of ''public interfaces'' often lends itself to testing in isolation ''without'' implementing special test logic (i.e. no stubbing required), which make the test implementation simpler. What's more, public APIs are most likely documented. As a result, ''non domain experts'' can more easily participate in the test implementation. Although API Testing often represents a smaller percentage of the software being exercised, the '''value''' and '''effort''' required is well understood.

== Behavior Testing ==
STRIDE also supports '''Behavior Testing''', which is different than Unit Testing or API Testing in that it does not focus on calling functions and validating return values. Behavior Testing, rather, validates the ''expected sequencing and state of the software'' executing under normal operating conditions. To learn more about the uniqueness of Behavior Testing [[What_is_Unique_About_STRIDE#STRIDE_includes_behavior-based_testing_techniques | '''read here''']]. We believe that Behavior Testing has a very high ''return-on-effort'' and can be easily deployed into legacy software systems.

Some of the advantages of '''Behavior Testing''' are:
* Tests are performed with fully functional software builds -- there are no code isolation issues challenges
* The technique is easy to grasp and is applicable to a large percentage of code bases
* Can execute with functional and black-box testing
* It's aasy to add [[Test Point | Test Points]] to provide coverage
* Others can easily participate in test implementation
* Tests can be written using [[Perl_Script_APIs#STRIDE::Test | scripting on the host]] and [[Expectation_Tests_in_C/C%2B%2B | native target code]]
* Validation is fully automated and thus does not rely on manual interpretation by domain experts
* Design knowledge is extracted and made available to other team members via [[Source Instrumentation Overview | instrumentation]]
* Very useful for continuous integration

[[Category: Overview]]

Types of Testing Supported by STRIDE

2010-06-11T22:28:38Z

Mikee: /* API Testing */

== Overview ==

One of the questions that should be asked is ''what is the '''value''' of the test?'' If the test does not discover any defects or does not provide ongoing regression, then the value is questionable. Also ''what is the '''effort''' in implementing the test?'' STRIDE has been uniquely designed to support maximizing the '' '''value''' '' of the test while minimizing the '' '''effort''' '' to implement it.

The STRIDE test system supports three general types of testing:
* '''Unit Testing'''
* '''API Testing'''
* '''Behavior Testing'''

== Unit Testing ==
[[Test Units Overview | STRIDE Unit Testing]] supports the model found in typical [http://en.wikipedia.org/wiki/XUnit xUnit-style] testing frameworks. STRIDE also offers a number of [[Test API | additional features]] optimized for testing embedded software. Traditional '''Unit Testing''' presents a number of challenges in testing embedded software:

* Testing functions/classes in '''isolation''' requires a lot of extra work, especially if your software was not designed upfront for testability
* Testing legacy software might have limited value, particularly if the software is stable with respect to defects
* The software is often not well suited for '''others''' to participate in the test implementation, since there is too much internal knowledge required to be productive
* It can be difficult to automate execution of the full set of tests on the real target device

== API Testing ==
STRIDE supports '''API Testing''' by leveraging the same [[Test Units Overview | techniques]] available for Unit Testing. API Testing differs from unit testing in that the tests focus on direct testing of a well-defined interface. This kind of testing is typically easy to scope depends on the size of the interface and often has a better ''return-on-effort''. The design of ''public interfaces'' often lends itself to testing in isolation ''without'' implementing special test logic (i.e. no stubbing required), which make the test implementation simpler. What's more, public APIs are most likely documented. As a result, ''non domain experts'' can more easily participate in the test implementation. Although API Testing often represents a smaller percentage of the software being exercised, the '''value''' and '''effort''' required is well understood.

== Behavior Testing ==
STRIDE supports '''Behavior Testing''' which is different than Unit Testing or API Testing in that it does not focus on calling functions and validating return values. Behavior Testing, rather, validates the ''expected sequencing of the software'' executing under normal operating conditions. To learn more about the uniqueness of Behavior Testing click [[What_is_Unique_About_STRIDE#STRIDE_includes_behavior-based_testing_techniques | '''here''']]. We believe that Behavior Testing has a very high ''return-on-effort'' and can be easily deployed into legacy software systems.

Some of the advantages of '''Behavior Testing''':
* Test with fully functional software builds (no isolation issues)
* Applies to a large percentage of the software
* Can execute with functional and black-box testing
* Easy to add [[Test Point | Test Points]] to provide coverage
* Others can easily participate in test implementation
* Tests can be written using [[Perl_Script_APIs#STRIDE::Test | scripting on the host]] and [[Expectation_Tests_in_C/C%2B%2B | native target code]]
* Validation is fully automated (not done manually by tribal experts)
* Design knowledge is extracted and made available to other team members via [[Source Instrumentation Overview | instrumentation]]
* Very useful for continuous integration

[[Category: Overview]]

Types of Testing Supported by STRIDE

2010-06-11T22:17:46Z

Mikee: /* Unit Testing */

== Overview ==

One of the questions that should be asked is ''what is the '''value''' of the test?'' If the test does not discover any defects or does not provide ongoing regression, then the value is questionable. Also ''what is the '''effort''' in implementing the test?'' STRIDE has been uniquely designed to support maximizing the '' '''value''' '' of the test while minimizing the '' '''effort''' '' to implement it.

The STRIDE test system supports three general types of testing:
* '''Unit Testing'''
* '''API Testing'''
* '''Behavior Testing'''

== Unit Testing ==
[[Test Units Overview | STRIDE Unit Testing]] supports the model found in typical [http://en.wikipedia.org/wiki/XUnit xUnit-style] testing frameworks. STRIDE also offers a number of [[Test API | additional features]] optimized for testing embedded software. Traditional '''Unit Testing''' presents a number of challenges in testing embedded software:

* Testing functions/classes in '''isolation''' requires a lot of extra work, especially if your software was not designed upfront for testability
* Testing legacy software might have limited value, particularly if the software is stable with respect to defects
* The software is often not well suited for '''others''' to participate in the test implementation, since there is too much internal knowledge required to be productive
* It can be difficult to automate execution of the full set of tests on the real target device

== API Testing ==
STRIDE supports '''API Testing''' leveraging the same [[Test Units Overview | techniques]] available for Unit Testing. The difference is concerning the focus of API Testing, which often has a better ''return-on-effort''. API Testing traditionally is about driving well-defined public interfaces. The design of ''public interfaces'' typically lends itself to testing in isolation ''without'' implementing special test logic (i.e. no stubbing required). Also public APIs are most likely documented. As a result, ''others'' can more easily participate in the test implementation. Although API Testing often represents a smaller percentage of the software being exercised, the '''value''' and '''effort''' required is well understood.

== Behavior Testing ==
STRIDE supports '''Behavior Testing''' which is different than Unit Testing or API Testing in that it does not focus on calling functions and validating return values. Behavior Testing, rather, validates the ''expected sequencing of the software'' executing under normal operating conditions. To learn more about the uniqueness of Behavior Testing click [[What_is_Unique_About_STRIDE#STRIDE_includes_behavior-based_testing_techniques | '''here''']]. We believe that Behavior Testing has a very high ''return-on-effort'' and can be easily deployed into legacy software systems.

Some of the advantages of '''Behavior Testing''':
* Test with fully functional software builds (no isolation issues)
* Applies to a large percentage of the software
* Can execute with functional and black-box testing
* Easy to add [[Test Point | Test Points]] to provide coverage
* Others can easily participate in test implementation
* Tests can be written using [[Perl_Script_APIs#STRIDE::Test | scripting on the host]] and [[Expectation_Tests_in_C/C%2B%2B | native target code]]
* Validation is fully automated (not done manually by tribal experts)
* Design knowledge is extracted and made available to other team members via [[Source Instrumentation Overview | instrumentation]]
* Very useful for continuous integration

[[Category: Overview]]

Types of Testing Supported by STRIDE

2010-06-11T22:11:51Z

Mikee:

== Overview ==

One of the questions that should be asked is ''what is the '''value''' of the test?'' If the test does not discover any defects or does not provide ongoing regression, then the value is questionable. Also ''what is the '''effort''' in implementing the test?'' STRIDE has been uniquely designed to support maximizing the '' '''value''' '' of the test while minimizing the '' '''effort''' '' to implement it.

The STRIDE test system supports three general types of testing:
* '''Unit Testing'''
* '''API Testing'''
* '''Behavior Testing'''

== Unit Testing ==
[[Test Units Overview | STRIDE Unit Testing]] supports the general model found in typical [http://en.wikipedia.org/wiki/XUnit xUnit-style] testing frameworks. STRIDE also offers a number of [[Test API | additional features]] optimized for testing embedded. But traditional '''Unit Testing''' does present a number of challenges for testing embedded software:

* Testing functions/classes in '''isolation''' requires a lot of extra work, especially if not designed upfront
* Testing legacy software might have very limited value (i.e. can't find any new defects)
* Typically not well suited for '''others''' to participate in the test implementation (too much internal knowledge required)
* Can be difficult to automate execution of the full set of tests on the real target device

== API Testing ==
STRIDE supports '''API Testing''' leveraging the same [[Test Units Overview | techniques]] available for Unit Testing. The difference is concerning the focus of API Testing, which often has a better ''return-on-effort''. API Testing traditionally is about driving well-defined public interfaces. The design of ''public interfaces'' typically lends itself to testing in isolation ''without'' implementing special test logic (i.e. no stubbing required). Also public APIs are most likely documented. As a result, ''others'' can more easily participate in the test implementation. Although API Testing often represents a smaller percentage of the software being exercised, the '''value''' and '''effort''' required is well understood.

== Behavior Testing ==
STRIDE supports '''Behavior Testing''' which is different than Unit Testing or API Testing in that it does not focus on calling functions and validating return values. Behavior Testing, rather, validates the ''expected sequencing of the software'' executing under normal operating conditions. To learn more about the uniqueness of Behavior Testing click [[What_is_Unique_About_STRIDE#STRIDE_includes_behavior-based_testing_techniques | '''here''']]. We believe that Behavior Testing has a very high ''return-on-effort'' and can be easily deployed into legacy software systems.

Some of the advantages of '''Behavior Testing''':
* Test with fully functional software builds (no isolation issues)
* Applies to a large percentage of the software
* Can execute with functional and black-box testing
* Easy to add [[Test Point | Test Points]] to provide coverage
* Others can easily participate in test implementation
* Tests can be written using [[Perl_Script_APIs#STRIDE::Test | scripting on the host]] and [[Expectation_Tests_in_C/C%2B%2B | native target code]]
* Validation is fully automated (not done manually by tribal experts)
* Design knowledge is extracted and made available to other team members via [[Source Instrumentation Overview | instrumentation]]
* Very useful for continuous integration

[[Category: Overview]]

Perl Script Snippets

2010-06-11T16:33:53Z

Mikee:

Below are some brief examples of script test modules that use the [[Perl Script APIs]] for STRIDE. Most of these examples are incomplete and are only intended to give a quick overview of how this API can be used.

With any perl module that you write, we recommend that you perform a quick syntax check before attempting to execute the module using the STRIDE runner. Syntax checking with warnings is easily accomplished by running with the <tt>-cw</tt> option, e.g.:

perl -cw MyTests.pm

This will compile the perl script without running it, and it will emit any syntax errors that are found.

== Canonical module format ==

<source lang="perl">
use strict;
use warnings;

package MyTests;
use base qw(STRIDE::Test);
use STRIDE::Test;

sub test_one : Test
{
ASSERT_TRUE(1);
}

sub test_two : Test
{
ASSERT_TRUE(0);
}

1;
</source>

== Subroutine attributes to declare test methods and fixtures ==

<source lang="perl">

sub a_test : Test {
# test method
}

sub startup_method : Test(startup) {
# startup fixturing
}

sub shutdown_method : Test(shutdown) {
# shutdown fixturing
}

sub setup_method : Test(setup) {
# setup fixturing
}

sub teardown_method : Test(teardown) {
# teardown fixture
}
</source>

== Test module including POD documentation ==

<source lang="perl">
package MyTests;
use base qw(STRIDE::Test);
use STRIDE::Test;

=head1 NAME

MyTests - example tests

=head1 DESCRIPTION

This is MyTests_1, a deeply funky piece of Perl code.

=head1 METHODS

=cut

=head2 test_one

this is a simple passing test.

=cut
sub test_one : Test
{
ASSERT_TRUE(1);
}

=head2 test_two

this is a simple passing test.

=cut
sub test_two : Test
{
ASSERT_TRUE(0);
}

1;
</source>

== Simple expectation test ==

<source lang="perl">
sub sync_exact : Test {
my $h = TestPointSetup(
ordered => 1,
expected => [
"point a",
"point b",
"point c"
],
unexpected => [ TEST_POINT_EVERYTHING_ELSE ]
);

#...start source under test, if necessary

# use Check if the events have all happened by the time
# you validate the test points. Otherwise use Wait with
# a reasonable timeout value.
$h->Check();
}
</source>

== Expectation test with predicate==

Here's how to specify a predicate for validating a test point. The predicate
shown is just a stub and doesn't do any actual validation.

<source lang="perl">
sub _myPredicate
{
my ($test_point, $expected_data) = @_;
# $test_point is a hashref with the following fields:
# label, data, size, bin, file, line
# use the test point data to perform validation and return
# nonzero value on success, 0 on failure.

return 1;
}

sub use_a_predicate : Test {
# use arrayref notation to specify label, count, predicate and expected data
# if you want to specify a predicate. Predicates can be specified for some, all
# or none of the test points. The fourth field is optional - it is just passed
# to the predicate as the second argument and is typically used to specify some
# expected data for the predicate to validate against.
my $h = TestPointSetup(
expected => [
["point a", 1, \&_myPredicate, 54321],
"point b",
["point c", 1, \&_myPredicate]
]
);

$h->Check();
}
</source>

== Reusing a trace data file as expectation ==

This examples shows a simple way to reuse captured trace data from a file as your expectation (this is an alternative to specifying the <tt>expected</tt> array directly). This example assumes the ''trace_data.yaml'' file lives in the same directory as the test module file, but you are free to change this - just change the <tt>expect_file</tt> argument accordingly.

<source lang="perl">
# these standard modules are only required for the file
# path logic used to generate a full file path for trace_file
use File::Basename;
use File::Spec;

sub trace_data : Test {
my $h = TestPointSetup(
ordered => 1,
expect_file => File::Spec->catfile(dirname(__FILE__), 'trace_data.yaml'),
);

# make any function calls necessary to start the SUT,
# then do a Check or Wait, depending on your scenario

$h->Check();
}
</source>

== Invoking a function on the target ==

=== Standalone two-line script for invoking a remote function ===
This simple script shows the minimal code required to make a remote function call of a function that has been enabled with [[Function Capturing#Remoting|STRIDE remoting]].
<source lang="perl">
use STRIDE;
$STRIDE::Functions->MyRemoteFunction();
</source>
More functions can be added to this script by making additional calls using the $STRIDE::Functions object. If your remote function accepts arguments, they should be added to the function call parameters (see the [[Perl_Script_APIs#STRIDE::Function|Perl API Reference]] for more information).

=== Making calls with test modules ===

Within test modules, remote functions can be easily invoked using the exported Functions object. The following two examples illustrate this.

=== Blocking syntax (blocks until function returns) ===

<source lang="perl">
my $retval = Functions->foo(1, "input string");
Functions->bar();
</source>

=== asynchronous execution (return immediately, function continues to execute on device) ===

<source lang="perl">
my $fh = Functions->{async}->foo(1, "input string");
my $retval = $fh->Wait(1000); #waits up to one second for the function to return
</source>

== Accessing compiler macro values (constants) ==
<source lang="c">
/* some compilation unit included in the STRIDE compilation process */
#define SOME_DEFINED_VALUE 42
</source>

<source lang="perl">
my $value = Constants->{SOME_DEFINED_VALUE};
</source>

[[Category:Tests in Script]]

Training Tests in Script

2010-06-10T21:16:11Z

Mikee: /* Can I use STRIDE test modules only for expectation testing ? */

== Background ==

The STRIDE Framework allows you to write expectation tests that execute on the host while connected to a running device that has been instrumented with [[Test Point|STRIDE Test Points]]. Host-based expectation tests leverage the power of scripting languages (perl is currently supported, others are expected in the future) to quickly and easily write validation logic for the test points on your system. What's more, since the test logic is implemented and executed on the host, your device software does not have to be rebuilt when you want to create new tests or change existing ones.

Please review the following reference articles before proceeding:

* [[Test Modules Overview|Scripting Overview]]
* [[Perl Script APIs|perl Test Modules]]

== What is an expectation test ? ==

An expectation test is a test that validates behavior by verifying the occurrence (or non-occurrence) of specific test points on your running device. The STRIDE Framework makes it easy to define expectation tests ''via a single setup API'' ([[Perl_Script_APIs#Methods|see TestPointSetup]]) . Once defined, expectation tests are executed by invoking a ''wait'' method that evaluates the test points on the device as they occur. The wait method typically blocks until the entire defined expectation set has been satisfied '''or''' until a timeout (optional) has been exceeded.

== How to I start my target processing scenario ? ==

It's often necessary, as part of test, to start the processing on the device that causes the test scenario to occur. Sometimes processing is invoked via external stimulus (e.g. send a command to a serial port or send a network message). Given the [http://search.cpan.org/ wealth of libraries] available for perl, it's likely that you'll be able to find modules to help you in automating common communication protocols.

If, on the other hand, the processing can be invoked by direct code paths in your application, you can consider using [[Function_Capturing|function fixturing]] via STRIDE. STRIDE function remoting allows you to specify a set of functions on the device that are to made available in the host scripting environment for remote execution. This can be a convenient way to expose device application hooks to the host scripting environment.

Whatever the approach, we strongly encourage test authors to find ways to minimize the amount of manual interaction required to execute expectation tests. Try to find ways to fully automate the interaction required to run your expectation scenarios. Fully automated tests are more likely to be run regularly and therefore provide a tighter feedback loop for your software quality.

== Can I use STRIDE test modules for any other testing besides expectation validation? ==

'''Yes''' - STRIDE test modules provide a language-specific way to harness test code. If you have other procedures that can be automated using perl code on the host, then you can certainly use STRIDE test modules to harness the test code. In doing so, you will get the reporting conveniences that test modules provide (like automatic POD doc extraction, etc. and suite/test case generation) - as well as unified reporting with your other STRIDE test cases.

== Sample: Expectations ==

For this training, we will again be using the sample code provided in the [[Expectations Sample]]. This sample demonstrates a number of common expectation test patterns as implemented in perl.

=== Sample Source: test_in_script/Expectations/s2_expectations_testmodules.pm ===

To begin, let's briefly examine the perl test module that implements the test logic. Open the file in your favorite editor (preferably one that supports perl syntax highlighting) and examine the source code along with a description that is provided at the beginning of each test. Here are some things to observe:

* The package name matches the file name -- ''this is required''
* We have included documentation for the module and test cases using standard POD formatting codes. As long as you follow the rules described [[Perl_Script_APIs#Documentation|here]], the STRIDE framework will automatically extract the POD during execution and annotate the report accordingly.
* Most of the test sequences are initiated via a remote function in the target app (<tt>Exp_DoStateChanges()</tt>). This function has been [[Function Capturing|captured]] using STRIDE and is therefore available for invocation using the Functions object in the test module. In one case, we also invoke the function asynchronously (see <tt>async_loose</tt>). Functions are invoke synchronously by default.
* In the <tt>check_data</tt> test, we validate integer values coming from the host that were passed as binary payloads to the test point. We use the perl [http://perldoc.perl.org/functions/pack.html pack] function to create a scalar value that matches the data expected from the target (target is the same as the host, in this case). If we were testing against a target with different integer characteristics (size, byte ordering), we would have to adjust our pack statement accordingly to produce a bit pattern that matched the target value(s). In many cases, this binary payload validation proves to be difficult to maintain and this is why ''we typically recommend using string data payloads'' on test points wherever possible.

=== Build the test app ===

So that we can run the sample, let's now build an off target test app that contains the source under tests -- you can follow the generic steps [[Off-Target_Test_App#Copy_Sample_Source|described here]]. '''Note:''' you can copy all of the sample files into the <tt>sample_src</tt> directory -- although only the source will be compiled into the app, this will make it easier to run the module.

=== Run the sample ===

Now launch the test app (if you have not already) and execute the runner with the following command:

<pre>
stride --device="TCP:localhost:8000" --database=../out/TestApp.sidb --run=s2_expectations_testmodule.pm
</pre>

(this assumes you are running from the <tt>sample_src</tt> directory of your off-target SDK. If that's not the case, you need to change the path to the database and test module files accordingly.)

If you'd like to see how log messages will appear in reports, you can add <tt>--log_level=all</tt> to this command.

=== Examine the results ===

The runner will produce a results (xml) file once the execution is complete. The file will (by default) have the same name as the database and be located in the current directory - so find the <tt>TestApp.xml</tt> file and open it with your browser. You can then use the buttons to expand the test suites and test cases. Here are some things to observe about the results:

* there is a single suite called '''s2_expectations_module'''. This matches the name given to the test module.
* the test module's suite has one annotation - it is a trace file containing all of the test points and logs that were reported to the host during the execution of the test module. This trace file can be useful if you want to get a sequential view of all test points that were encountered during the execution of the module.
* the test module suite contains 13 test cases -- each one corresponds to a single test case (test function) in the module. The description for each test module was automatically generated from the POD documentation that was included in the test module file.
* each test case has a list of several annotations. The first is a simple HTML view of the source code of the test itself. This can be useful for quickly inspecting the code that was used to run the test without having to go to your actual test module implementation file. The remaining annotation contain information about each test point that was hit during processing and any expectation failures or timeouts that were encountered.

[[Category: Training]]

Training Overview (old)

2010-06-07T21:06:25Z

Mikee: moved ++WIP++ Training Overview to Training Overview

Our training approach is based on articles (here in the wiki) and on a set of code samples that readily execute in our off-target (desktop) environment. Our training focuses on a self-guided tour of the product using the samples we provide as the primary study material. Please review the sections below before proceeding to the specific training topics.

== Prerequisites ==

In order to build and execute the samples that we use in the training, please complete or verify the following prerequisites

# complete the [[Desktop Installation]]. Be sure to include perl in the installation if you want to do training related to host-based test scripting. Verify that the diagnostics pass after installation.
# install the desktop development toolchain for your host, as described for [[Desktop_Installation#GCC|linux]] or [[Desktop_Installation#Microsoft_Visual_Studio|windows]]. This is required for building the STRIDE runtime and samples.
# read the [[STRIDE Overview]] article to familiarize yourself with the high-level approach and components of STRIDE.

== How we train ==

Our training articles are based on a handful of samples that we provide with the off-target framework distribution. The samples are usually self-documented (using doxygen or perldoc) and this content will be attached to the test report whenever a sample is executed.

The samples were created to be as simple as possible while sufficiently demonstrating the topic at hand. In particular, the samples are '''very light''' on core application logic (that is, the source under test) -- they focus instead on the code that leverages STRIDE to define and execute tests. As you review the sample code, if you find yourself confused about which code is the test logic and which is the source under test, try reading the source file comments to discern this. If you are still unclear about how the source is organized, feel free to contact us for clarification.

== What you need to do ==

In order to get the full benefit from these training articles, we recommend you do the following:

* follow the wiki links we provide in the training articles. These links provide rich technical information on the topics covered by the training. These are also articles you will likely refer to in the future when you are implementing your own tests.
* read/review all sample source code prior to running. The samples consist almost entirely of source code, so it makes sense to use a source code editor (one you are familiar with) for this purpose.
* build and execute the samples using the off-target framework. If you completed your installation as instructed above, it should be fully functional when you do the training.
* review the reports that are produced when you run the samples. The reports give you a feel for how data is reported in the STRIDE Framework. The extracted documentation is also provided in the report.
* For most samples, we provide some observations that help summarize aspects of the results that might be of interest to you. These observations are not necessarily comprehensive - in fact, we hope you'll discover other interesting features in the samples that we haven't mentioned.

Training Overview (old)

2010-06-07T21:06:01Z

Mikee:

Training Overview (old)

2010-06-07T20:47:46Z

Mikee:

Our training approach is based on articles (here in the wiki) and on a set of code samples that readily execute in our off-target (desktop) environment. Our training focuses on a self-guided tour of the product using the samples we provide as the primary study material. Please review the sections below before proceeding to the specific training topics.

== Prerequisites ==

In order to build and execute the samples that we use in the training, please complete of verify the following prerequisites

# complete the [[Desktop Installation]]. Be sure to include perl in the installation if you want to do training related to host-based test scripting and verify the diagnostics.
# install the development toolchain for your host, as described for [[Desktop_Installation#GCC|linux]] or [[Desktop_Installation#Microsoft_Visual_Studio|windows]]. This is required for building the STRIDE runtime and samples.
# read the [[STRIDE Overview]] article to familiarize yourself with the high-level approach and components of STRIDE.

== What we train on ==

Our training articles are based on a handful of samples that we provide with the off-target framework distribution. The samples are usually self-documented using doxygen, the contents of which will be attached to the test report whenever a sample is executed.

The samples were created to be as simple as possible while still adequately demonstrating the topic at hand. In particular, the samples are '''very light''' on core application logic (that is, source under test) -- they focus instead on the code that leverages STRIDE to define and execute tests. As you review the sample code, if you find yourself confused about which code is the test logic and which is the source under test, try reading the source file comments to discern them. If you are still unclear about how the source is organized, feel free to contact us for clarification.

== What you need to do ==

In order to get the full benefit from these training articles, we recommend you do the following:

* follow the wiki links we provide in the training articles. These links provide rich technical information on the topics covered by the training. These are also articles you will likely refer to in the future when you are implementing your own tests.
* read/review all sample source code prior to running. The samples consist almost entirely of source code, so it makes sense to use a source code editor (one you are familiar with) for this purpose.
* build and execute the samples using the off-target framework. If you completed your installation as instructed above, it should be fully functional when you do the training.
* review the reports that are produced when you run the samples. The reports give you a feel for how data is reported in the STRIDE Framework. The extracted documentation is also provided in the report.
* For most samples, we provide some observations that help summarize aspects of the results that might be of interest to you. These observations are not necessarily comprehensive - in fact, we hope you'll discover other interesting features in the samples that we haven't mentioned.

2010-06-01T17:08:34Z

Mikee: moved ++WIP++ Training Instrumentation to Training Instrumentation

== Background ==

Source instrumentation is the means by which application domain experts strategically instrument the source under test so as to enable themselves or others to write expectation tests against the running code. Source instrumentation is one of the first steps toward enabling expectation testing of your application.

Source instrumentation is accomplished by a set of simple yet powerful macros provided by the STRIDE Runtime library. The macros are easily activated/deactivated for any build through the use of a single preprocessor macro value. When activated, they provide a means to validate the behavior of your application as it is running.

Please review the following reference articles before proceeding:

* [[Source Instrumentation Overview|Instrumentation Overview]]
* [[Test Point|Test Points]]
* [[Test Log|Test Logs]]

== How Do I Instrument ? ==

As described in [[Source Instrumentation Overview|the overview]], you can begin instrumenting your source code by including the <tt>srtest.h</tt> header file and then adding '''srTEST_POINT*''' and '''srTEST_LOG*''' in source locations of interest. These macros will be inactive (no-op) unless '''STRIDE_ENABLED''' is defined during compilation.

== Where Should I Instrument ? ==

When thinking about ''where'' to instrument your source under test, consider these high-value locations:

* Function entry and exit points. Include call parameters as data if appropriate.
* State transitions
* Data transitions (i.e. any point where important data changes value)
* Error conditions
* Callback functions

In addition, when you start to instrument your source code, it's beneficial to pause and consider some of the test cases you expect to validate against the test points you are inserting. For each potential test case, you might also want to consider some of the characteristics you'd use for those tests, as described in [[Expectations|this article]]. The characteristics you expect to apply for various test cases might, for example, inform things like how you label your test points or what kind of data you include.

== What's The Difference between a Test Point and a Test Log ? ==

[[Test Point|Test Points]] can be used for validation since they are what's checked when you run a STRIDE expectation test. Test Logs, on the other hand, are purely informational and will be included in the report, according to the log level indicated when the [[Stride_Runner#Options|STRIDE Runner]] was executed. Refer to the [[#Background|background links above]] for more information on each of these instrumentation types.

== What About Data ? ==

Including data with your test points adds another level of power to the validation of your source. Here are some general recommendations for using data effectively:

* Try to use string data payloads wherever possible. String payloads are considerably more human-friendly when viewing test results and they allow for relatively simple string comparison validation.
* If you need to do complex validation of '''multi-field data''' in a test point, consider using an object serialization format such as [http://json.org/ JSON]. Standard formats like this are readily parsable in host scripting languages. If, however, you will ''only'' be writing expectation tests in native code for execution on the target, then string serialization formats might be too cumbersome for validation. In that case, using binary payloads (structures, typically) is reasonable.
* The data payloads for a test point are limited to a fixed size (512 bytes by default, but configurable if needed). If you have large data payloads that you need to validate and you are using host-based script validation logic, consider using [[File Transfer Services]] to read/write the data as files on the host. If, on the other hand, you are using native code for validation, you can independently manage your own buffers of data (heap allocated, for example) for validation and use the test point payloads only to transmit addresses and sizes of the payloads.

== Sample Code: s2_expectations_source ==

For this training, we will simply review some existing sample source code and explain the motivation behind some of the instrumentation. In particular, we will peruse the source code associated with the [[Expectations Sample]].

=== Samples/test_in_script/Expectations/s2_expectations_source.c ===

This is the source under test for a simple example that demonstrates expectation tests where the test logic is written in a script module (perl) that runs on the host. The source under test is a simple state machine and we have chosen to instrument each of the 4 states with one or more test points. Open the source file in your preferred editor and search for '''srTEST_POINT'''. You will see that we have test points in the following functions:

* ''SetNewState()'': This is a shared function that is called to effectuate a state change. The test point label is just a static string chosen by the instrumenter and the name of the new state is included as string data on the test point.
* ''Start()'': The Start state includes a single test point with no data. The label is gotten from a shard function that returns a string representation for any given state (''GetStateName()'').
* ''Idle()'': The Idle state records a single test point. The test point label is again obtained from ''GetStateName()'' and the data associated with the test point is a transition count value that the software under test is maintaining. This state function also includes an ''info'' level test log. Since it is an info level log, it will not be captured during testing unless you explicitly set the log level to ''info'' or higher when executing the tests.
* ''Active()'': The Active state has three distinct test points. The first point is similar to previous states and records a test point with the name of the current state as a label and transition count as data. The second test point shows another example of including string data in a test point. The third test point shows an example of simple JSON serialization to include several values in a single test point payload.
* ''Exp_DoStateChanges()'': This function drives the state transitions and includes one warning log message that is not hit during normal execution of the code.
* ''End()'': The End state records a single test point with the state name as label, similar to the other states already mentioned.

=== Build, Run, and Trace ===

So that you can see the trace points and logs in action, we will now build an off-target test app with this sample source included. Follow the [[Off-Target_Test_App#Build_Steps|instructions here]], using the same source files from the Expectation sample that we just reviewed.

Now we want to invoke the function from our source under test using the [[STRIDE Runner]] and request that the runner show a trace of any trace points that occur in the test app during execution.

In order to invoke the function that starts our state transitions, let's create the following two line perl script (use your favorite editor):

<source lang="perl">
use STRIDE;
$STRIDE::Functions->Exp_DoStateChanges();
</source>

Save this file as <tt>do_state_changes.pl</tt> in the <tt>src</tt> directory of your SDK. Now open a command prompt and change to that same directory. Execute the stride runner:

<pre>
stride --device="TCP:localhost:8000" --database="../out/TestApp.sidb" --run=do_state_changes.pl --trace
</pre>

This tells the runner to execute our script ''and'' to report any test points that are encountered on the device (or our test app, in this case) during the execution of that script. You should see a number of test points reported while the script executes and then some summary information about the tests that were executed (there are no tests executed in this case).

Now let's include any log messages by specifying the <tt>--log_level</tt> flag:

<pre>
stride --device="TCP:localhost:8000" --database="../out/TestApp.sidb" --run=do_state_changes.pl --trace --log_level=all
</pre>

When you run this, you should see the same output as before as well as one ''LOG'' entry. If we had many LOG entries and wanted to filter based on log_level, we would change the value passed to <tt>--log_level</tt> accordingly.

== What next ? ==

Now you've seen how instrumentation is strategically placed in source under test and can be traced during execution under the STRIDE Runner. We recommend that you proceed to some of our other training topics to learn how to create tests on the host (in script) or on the target (in native code) that use your instrumentation test points as a means for validation.

[[Category: Training]]

2010-05-28T22:31:47Z

Mikee:

== Test Points ==

Refer to [[Test Point Testing in C/C++|this article]].

== Test Fixturing ==

Refer to [[Test Fixturing in C/C++|this article]].

== Test Documentation ==

Refer to [[Test Documentation in C/C++|this article]].

== Runtime Test Services ==

Refer to [[Runtime Test Services|this article]] for a complete listing of the Runtime Test Services APIs.

== File Transfer Services ==

Refer to [[File Transfer Services|this article]] for a complete listing of the File Transfer Services APIs.

== Test Doubles ==

Refer to [[Using Test Doubles|this article]].

[[Category:Test Units]]