What is Unique About STRIDE: Difference between revisions

From STRIDE Wiki
Jump to navigation Jump to search
No edit summary
 
(19 intermediate revisions by 3 users not shown)
Line 1: Line 1:
= STRIDE is a cross-platform framework =
STRIDE was designed by embedded engineers to facilitate implementing and executing '''On-Target White-box Testing'''. The various distinctive features describe here make it easy for both developers and testers to participate in validating the software during the development phase. The following is a high-level list of some of the unique features of STRIDE:
The [[Runtime_Reference | '''STRIDE Runtime''']] is written in standard C on top of a simple [[Platform_Abstraction_Layer | '''platform abstraction layer''']] that enables it to work on virtually any target platform. It is delivered as source code to be included in the application's build system. STRIDE also auto-generates [[Intercept_Module | '''harnessing and remoting logic''']] as source code during the make process, removing any dependencies on specific compilers and / or processors. The transport between the host and target is configurable, supporting serial, USB, and TCP/IP. Custom transports are also available. STRIDE is configured for single process application by default, but also supports multiple process applications (i.e. Linux, Windows CE, etc.). Testing can also be conducted using an [[Off-Target_Environment | '''Off-Target environment''']] such as Windows and Linux host machines.
<br>
<br>


The cross-platform framework facilitates a unified testing approach for all team members, enabling organizations to standardize on a test workflow independent of the target platform being used or what branch of software is being changed.  
== STRIDE works on virtually any target platform ==
STRIDE's cross-platform framework facilitates a ''unified testing approach'' for all team members, enabling organizations to standardize on a test workflow that is independent of the target platform being used or what branch of software is being changed.


= STRIDE software builds are both functional and testable =
=== Runtime written in standard C ===
STRIDE enables software builds to be both fully ''functional'' and ''testable'' at the same time. The software works exactly the same as before. Whatever the software image was used for in the past -- system testing, developer debugging, etc. -- is still applicable. The STRIDE [[Test_Units | '''test logic''']] is separated from the application source code and is NOT executed unless invoked via the [[Stride_Runner | '''runner''']]. The [[Source_Instrumentation_Overview | '''source instrumentation''']] is only active when executing tests. The impact of built-in testability to the software application is nominal. The application can easily be switched back to a ''non-testable'' build by simply removing the [[Runtime_Integration#STRIDE_Feature_Control | '''STRIDE_ENABLED''']] preprocessor directive. This automatically controls all STRIDE related source code and macros. There are no changes required to the build process to enable or disable this functionality.  
The [[Runtime_Reference | '''STRIDE Runtime''']] is written in standard C on top of a simple [[Platform_Abstraction_Layer | '''platform abstraction layer''']] that enables it to work across platforms. It can be configured for a single process multi-threading environment or multiple process environments (i.e. Linux, Windows, Embedded RTOS, etc.). It is delivered as source code to be included in the application's build system. The transport between the host and target is configurable and supports Serial and TCP/IP by default. Custom transports are also available. The runtime has also been tailored specifically to embedded applications, overhead is minimal. It consumes very little memory for table and control block storage. [[Frequently_Asked_Questions_About_STRIDE#What_is_the_size_of_the_STRIDE_Runtime.3F | Resource usage]] is configurable and can be tailored to the limitations of the target platform.


The built-in ''testability'' is similar in concept to a ''debug build'' except the focus is on testing. The ''testable build'' leverages the existing software build process by integrating into the same ''make system'' used by the development team (no one-off or special builds). Automatically included in the build is test automation controllable from the host. When tests are executed timing analysis is also included with the generated [[Reporting_Model | '''test report''']].
===Integrates With Your Existing Build System ===
Developers can pre-flight test their source code changes before ''committing'' them to a baseline. Builds can be [[Setting_up_your_CI_Environment | '''automatically regression''']] tested as part of the daily build process.  
STRIDE also auto-generates [[Intercept_Module | '''harnessing and remoting logic''']] as source code during the make process, removing any dependencies on specific compilers and / or processors.


= STRIDE facilitates deeper API/Unit Testing =
===Supports Off-Target Testing ===
STRIDE offers unique techniques that can be leveraged for deeper API/Unit test coverage. STRIDE provides support for this type of testing [[Test_Units_Overview | '''in C/C++''']]. There are no special APIs required to register tests, suites, etc. Just write your test in any combination of C/C++ and the auto-generated [[Intercept_Module | '''intercept module''']] via the [[Build_Tools | '''STRIDE build tools''']] takes care of everything. Also aggregating tests from separate teams is supported -- no coordination is required. Writing API/Unit tests in native code is the simplest way to begin validating the software. There is no new language to learn, no proprietary editor, and the same workflow used when programming can be leveraged. This type of testing works well for:
Testing can also be conducted using an [[Off-Target_Environment | '''Off-Target Environment''']] which is provided for desktop (Windows, Linux or FreeBSD) host machines. The [[Posix SDK | '''Posix''']] and [[Windows SDK | '''Windows''']] ''SDKs'' allow for a seamless transition between the real target and an Off-Target host environment.


* calling APIs directly
== STRIDE provides built-in automation and reporting ==
* validating C++ classes
STRIDE enables software builds to be both fully ''functional'' and ''testable'' at the same time. Built-in ''automation and reporting'' is similar in concept to a ''debug build'' except the focus is on testing.
* isolating modules
===Testable Builds===
* critical processing / timing
The ''testable build'' leverages the existing software build process by integrating into the same ''make system'' used by the development team (no one-off or special builds). Automatically included in the build is test automation controllable from the host. When tests are executed, an xml is generated on the host (and optionally uploaded to Test Space) which includes detailed test results and timing analysis. This xml file uses a custom schema for representing the [[Reporting_Model | '''hierarchy of results''']] (suites, cases, etc.). Also included is a stylsheet specification (which will be written to the same directory as the xml file) that allows the results to be viewed as HTML in a browser.
* and much more ...
 
Developers can easily pre-flight test their source code changes before ''committing'' them to a baseline. Builds can be [[Setting_up_your_CI_Environment | '''automatically regression tested''']] as part of the daily build process.
 
===Functional Builds===
The ''testable software'' is still fully functional and works exactly the same as before. Whatever the software image was used for in the past -- system testing, developer debugging, etc. -- is still applicable. The STRIDE [[Test_Units | '''native test logic''']] is separated from the application source code and is NOT executed unless invoked via the [[Stride_Runner | '''runner''']]. The [[Source_Instrumentation_Overview | '''source instrumentation''']] is only active when executing tests. The impact of built-in testability to the software application is nominal.
 
The application can easily be switched back to a ''non-testable'' build by simply removing the [[Runtime_Integration#STRIDE_Feature_Control | '''STRIDE_ENABLED''']] preprocessor directive and rebuilding. This flag controls all STRIDE related source code and macros; there are no changes required to the build process to enable or disable this functionality.


For more advanced testing scenarios, dependencies can be [[Using_Test_Doubles | '''doubled''']]. This feature provides a means for intercepting C/C++ language global functions on the target and substitute a stub, fake, or mock. The substitution is all controllable via the [[Stride_Runner | '''runner''']], allowing the software to continue executing normally when not running a test.  
== STRIDE offers testing techniques for deeper coverage ==
STRIDE offers numerous testing techniques that enable deeper and more effective testing with less effort.
===Test Macros===
[[Test_Macros | Test Macros in native code]] provide one-line shortcuts for validating assertions and automatic report annotation in the case of failures. Test Macros are supported in both C/C++ and [[Perl_Script_APIs#Assertions | our scripting solution]].


[[File_Transfer_Services | ''' File fixturing''']] is another technique that can be leveraged to drive better testing. Test code executing on the target platform can perform file operations on the host remotely. This enables opening, reading, writing, etc. files while executing target-based test logic. File fixturing allows bypassing external input into the system for more controlled and isolated testing.  
===Fixturing===
Setting up your ''Software Under Test'' to be in the right state for a test is critical for repeatability. STRIDE supports a set of [http://en.wikipedia.org/wiki/Test_fixture '''Test fixture techniques'''] that include
* [[Test_Fixturing_in_C/C%2B%2B#Specifying___Fixturing_Methods | '''Setup/Teardown''']]
* [[Parameterized_Test_Units | '''Parameter passing to tests''']]
* [[File_Transfer_Services | '''Host/Target file transfer''']]
* [[Function_Capturing | '''Function remoting''']]
* etc.  


There are numerous other features that can be leveraged to facilitate deeper API/Unit test coverage:
These techniques encourage best-practices such as partitioning of fixturing code and actual test code, and--in addition--these same techniques can be leveraged to dynamically configure your tests at run-time.
* [[Test_Macros | '''Assertion macros''']]
* Leveraging [[Test Point Testing in C/C++ | '''Test Points''']] for behavior testing
* Passing parameters to Test Units
* Seamless publishing to [[STRIDE_Test_Space | '''Test Space''']]
* and much [[Test_API | ''' more ... ''']]


= STRIDE includes behavior-based testing techniques =
===Expectations===
STRIDE leverages [[Source_Instrumentation_Overview | '''source instrumentation''']] to provide '' '''behavior-based testing techniques''' '' that can be applied to the executing software application. The execution sequencing of the code, along with data, can be automatically validated based on [[Expectations | '''expectations''']]. Behavior-based testing is different than unit testing or API testing. Behavior testing does not focus on calling functions / methods and validating their return values. Behavior testing validates the ''expected sequencing of the software'' executing under normal operating conditions. This can span threads and process boundaries, and even multiple targets, as the application(s) is running. Leveraging macros -- called [[Test_Point | '''Test Points''']] -- domain experts strategically instrument the source under test. Data can also optionally be associated with a ''' ''Test Point'' ''', which is often critical to validating that the software is behaving as expected. This type of validation can be applied to a wide-range of testing scenerios:
STRIDE leverages [[Source_Instrumentation_Overview | '''source instrumentation''']] to provide '' [[Expectations | '''Expectations''']] ''as an additional validation technique that can be applied to the executing software application. The execution sequencing of the code, along with state data, can be automatically validated based on ''what is expected''. This validation technique does not focus on calling functions / methods but rather verifies ''code sequencing''. This can span threads, process boundaries, and even multiple targets, as the application(s) is running. Leveraging simple macros -- called [[Test_Point | '''Test Points''']] -- developers strategically instrument the source under test.


The test validation can be implemented in both [[Expectation_Tests_in_C/C%2B%2B | '''C/C++ on the target''']] and [[Perl_Script_APIs#STRIDE::Test | '''Perl script on the host''']]. In either case, the validation is done without impacting the application's performance (the on-target test code is executed in a background thread and scripts are executed on the host machine). When failures do occur, context is provided with the file name and associated line number of the failed expectations. This type of validation can be applied to a wide-range of testing scenarios:
* State Machines
* State Machines
* Data flow through system components
* Data flow through system components
* Sequencing between threads
* Sequencing between threads
* Drivers that don't return values
* Drivers that don't return values
* and much more ..
* and much more ...
 
''Software Quality Assurance (SQA)'' can also leverage [[Expectations | '''Expectations''']] as part of their existing functional / system testing. Because the [[Stride_Runner | '''runner''']] is a command line utility, it is easily controlled from existing test infrastructure. SQA can use the  [[Source_Instrumentation_Overview | '''instrumentation''']] to create their own [[Reporting_Model#Suites | '''test suite''']] using scripting that executes in concert with existing test automation.
 
===Test Doubles===
For more advanced testing scenarios, dependencies can be [[Test_Double | '''doubled''']]. This feature provides a means for intercepting C/C++ global functions on the target and substituting a stub, fake, or mock. The substitution is all controllable via the runtime, allowing the software to continue executing normally when not running a test.


Another key element to STRIDE based ''behavior testing'' is that the expected code sequencing is automatically validated. This automatic validation can be used for on-going regression testing. The expectations are realized by simple table definitions along with hooks for customization related to optional data. When failures do occur, context is provided with the file name and associated line number of the unexpected '''Test Point(s)''' behavior. The test validation can be implemented in both [[Expectation_Tests_in_C/C%2B%2B | '''native target code''']] and [[Perl_Script_APIs#STRIDE::Test | '''scripting on the host''']]. In either case, the validation is done without impacting the application's performance (i.e. target code executed in a background thread and host script executed on desktop).
===Other Features===
There are numerous other features that can be leveraged to facilitate deeper test coverage:
* Remoting global functions for script based API testing
* Built-in logging on test execution
* Dynamic test / suite creation
* Seamless publishing to [[STRIDE_Test_Space | '''Test Space''']]
* and much [[Test_API | ''' more ... ''']]


Software Quality Assurance(SQA) can also add behavior-based testing as part of their existing functional / system testing. Because the [[Stride_Runner | '''runner''']] is a command line utility, it is easily controlled from existing test infrastructure. SQA can also leverage the '' '''instrumentation''' '' to create their own [[Reporting_Model#Suites | '''test suites''']] using scripting that executes in concert with existing test automation.
== STRIDE supports test implementation in C/C++ and Script ==
The test validation can be implemented in both ''native code'' on the target and ''script'' on the host.
 
===Tests in C/C++===
Writing API/Unit tests in [[Test_Units_Overview | '''native code''']] is the simplest way to begin validating the software. There is no new language to learn, no proprietary editor, and your normal programming workflow is not interrupted. Also there is no special APIs required to register tests, suites, etc. Just write your test in any combination of C/C++ and the auto-generated [[Intercept_Module | '''intercept module''']] via the [[Build_Tools | '''STRIDE build tools''']] takes care of everything. Tests from separate teams are automatically aggregated by the system -- no coordination is required.
 
This type of testing works well for:
 
* Calling APIs directly
* Validating C++ classes
* Isolating modules
* Critical processing / timing
* and much more...
===Tests in Script===
STRIDE also supports writing tests in [[Test_Modules_Overview | '''Perl script''']]. When writing test scripts there are minimal dependencies on the software build process. Tests can also validate global functions, setup conditions from the host, etc.
 
Scripts are well suited for Integration Testing focusing on:
 
* State Machines
* Data flow through system components
* Sequencing between threads
* and much more...


= STRIDE optimizes failure resolution =
In either case, the validation is done without impacting the application's performance.
When executing '' '''behavior-based tests''' '' or '' '''API/unit tests''' '', results can be automatically uploaded to [[STRIDE_Test_Space | '''STRIDE Test Space''']] for storing and analyzing.  Result data is uploaded manually (using the web interface) or automatically using the [[Stride_Runner | '''runner''']]. Hosting test results in a ''central location'' with easy access via a browser enables the entire team to better participate in ensuring quality as apart of the development process. [[STRIDE_Test_Space#Results_View | '''Reports''']] containing test results, timing, [[Tracing | '''tracing logs''']], and built-in test documentation (supported in both [[Test_API#Test_Documentation | '''target code''']] and [[Perl_Script_APIs#Documentation | '''script''']]) all correlated together enhances the context for all team members to manage testing and optimize resolving failures. Team collaboration on testing activities is also facilitated with auto-generated email [[Notifications | '''notifications''']] and [[STRIDE_Test_Space#Messages | '''messaging''']]. Emails contain links to test failures where file names and line numbers are provided to aid in resolution. Messaging allows team members to more effectively communicate on the specifics of test results while minimizing information required in traditional emails, status reports, etc.  


== STRIDE includes Web-based test results management ==
When executing '''tests''', results can be uploaded to [[STRIDE_Test_Space | '''STRIDE Test Space''']] for persistence and analysis. Test result data can be uploaded manually (using the web interface) or automatically using the [[Stride_Runner | '''Runner''']].
===Centralized Hosting===
Hosting test results in a ''central location'' with easy access via a browser enables the entire team to better participate in ensuring quality during the ongoing development process. [[STRIDE_Test_Space#Results_View | '''Reports''']] containing test results, timing, [[Tracing | '''tracing logs''']], and built-in test documentation (supported in both [[Test_API#Test_Documentation | '''C/C++''']] and [[Perl_Script_APIs#Documentation | '''Perl''']]) all correlated together enhances the context for all team members to manage testing and optimize resolving failures.
===Team Collaboration===
Team collaboration on testing activities is also facilitated with auto-generated email [[Notifications | '''notifications''']] and [[STRIDE_Test_Space#Messages | '''messaging''']]. Emails contain links to test failures where file names and line numbers are provided to aid in resolution. Messaging allows team members to more effectively communicate on the specifics of test results while minimizing information required in traditional emails, status reports, etc.


[[Category: Overview]]
[[Category: Overview]]

Latest revision as of 15:17, 7 July 2015

STRIDE was designed by embedded engineers to facilitate implementing and executing On-Target White-box Testing. The various distinctive features describe here make it easy for both developers and testers to participate in validating the software during the development phase. The following is a high-level list of some of the unique features of STRIDE:

STRIDE works on virtually any target platform

STRIDE's cross-platform framework facilitates a unified testing approach for all team members, enabling organizations to standardize on a test workflow that is independent of the target platform being used or what branch of software is being changed.

Runtime written in standard C

The STRIDE Runtime is written in standard C on top of a simple platform abstraction layer that enables it to work across platforms. It can be configured for a single process multi-threading environment or multiple process environments (i.e. Linux, Windows, Embedded RTOS, etc.). It is delivered as source code to be included in the application's build system. The transport between the host and target is configurable and supports Serial and TCP/IP by default. Custom transports are also available. The runtime has also been tailored specifically to embedded applications, overhead is minimal. It consumes very little memory for table and control block storage. Resource usage is configurable and can be tailored to the limitations of the target platform.

Integrates With Your Existing Build System

STRIDE also auto-generates harnessing and remoting logic as source code during the make process, removing any dependencies on specific compilers and / or processors.

Supports Off-Target Testing

Testing can also be conducted using an Off-Target Environment which is provided for desktop (Windows, Linux or FreeBSD) host machines. The Posix and Windows SDKs allow for a seamless transition between the real target and an Off-Target host environment.

STRIDE provides built-in automation and reporting

STRIDE enables software builds to be both fully functional and testable at the same time. Built-in automation and reporting is similar in concept to a debug build except the focus is on testing.

Testable Builds

The testable build leverages the existing software build process by integrating into the same make system used by the development team (no one-off or special builds). Automatically included in the build is test automation controllable from the host. When tests are executed, an xml is generated on the host (and optionally uploaded to Test Space) which includes detailed test results and timing analysis. This xml file uses a custom schema for representing the hierarchy of results (suites, cases, etc.). Also included is a stylsheet specification (which will be written to the same directory as the xml file) that allows the results to be viewed as HTML in a browser.

Developers can easily pre-flight test their source code changes before committing them to a baseline. Builds can be automatically regression tested as part of the daily build process.

Functional Builds

The testable software is still fully functional and works exactly the same as before. Whatever the software image was used for in the past -- system testing, developer debugging, etc. -- is still applicable. The STRIDE native test logic is separated from the application source code and is NOT executed unless invoked via the runner. The source instrumentation is only active when executing tests. The impact of built-in testability to the software application is nominal.

The application can easily be switched back to a non-testable build by simply removing the STRIDE_ENABLED preprocessor directive and rebuilding. This flag controls all STRIDE related source code and macros; there are no changes required to the build process to enable or disable this functionality.

STRIDE offers testing techniques for deeper coverage

STRIDE offers numerous testing techniques that enable deeper and more effective testing with less effort.

Test Macros

Test Macros in native code provide one-line shortcuts for validating assertions and automatic report annotation in the case of failures. Test Macros are supported in both C/C++ and our scripting solution.

Fixturing

Setting up your Software Under Test to be in the right state for a test is critical for repeatability. STRIDE supports a set of Test fixture techniques that include

These techniques encourage best-practices such as partitioning of fixturing code and actual test code, and--in addition--these same techniques can be leveraged to dynamically configure your tests at run-time.

Expectations

STRIDE leverages source instrumentation to provide Expectations as an additional validation technique that can be applied to the executing software application. The execution sequencing of the code, along with state data, can be automatically validated based on what is expected. This validation technique does not focus on calling functions / methods but rather verifies code sequencing. This can span threads, process boundaries, and even multiple targets, as the application(s) is running. Leveraging simple macros -- called Test Points -- developers strategically instrument the source under test.

The test validation can be implemented in both C/C++ on the target and Perl script on the host. In either case, the validation is done without impacting the application's performance (the on-target test code is executed in a background thread and scripts are executed on the host machine). When failures do occur, context is provided with the file name and associated line number of the failed expectations. This type of validation can be applied to a wide-range of testing scenarios:

  • State Machines
  • Data flow through system components
  • Sequencing between threads
  • Drivers that don't return values
  • and much more ...

Software Quality Assurance (SQA) can also leverage Expectations as part of their existing functional / system testing. Because the runner is a command line utility, it is easily controlled from existing test infrastructure. SQA can use the instrumentation to create their own test suite using scripting that executes in concert with existing test automation.

Test Doubles

For more advanced testing scenarios, dependencies can be doubled. This feature provides a means for intercepting C/C++ global functions on the target and substituting a stub, fake, or mock. The substitution is all controllable via the runtime, allowing the software to continue executing normally when not running a test.

Other Features

There are numerous other features that can be leveraged to facilitate deeper test coverage:

  • Remoting global functions for script based API testing
  • Built-in logging on test execution
  • Dynamic test / suite creation
  • Seamless publishing to Test Space
  • and much more ...

STRIDE supports test implementation in C/C++ and Script

The test validation can be implemented in both native code on the target and script on the host.

Tests in C/C++

Writing API/Unit tests in native code is the simplest way to begin validating the software. There is no new language to learn, no proprietary editor, and your normal programming workflow is not interrupted. Also there is no special APIs required to register tests, suites, etc. Just write your test in any combination of C/C++ and the auto-generated intercept module via the STRIDE build tools takes care of everything. Tests from separate teams are automatically aggregated by the system -- no coordination is required.

This type of testing works well for:

  • Calling APIs directly
  • Validating C++ classes
  • Isolating modules
  • Critical processing / timing
  • and much more...

Tests in Script

STRIDE also supports writing tests in Perl script. When writing test scripts there are minimal dependencies on the software build process. Tests can also validate global functions, setup conditions from the host, etc.

Scripts are well suited for Integration Testing focusing on:

  • State Machines
  • Data flow through system components
  • Sequencing between threads
  • and much more...

In either case, the validation is done without impacting the application's performance.

STRIDE includes Web-based test results management

When executing tests, results can be uploaded to STRIDE Test Space for persistence and analysis. Test result data can be uploaded manually (using the web interface) or automatically using the Runner.

Centralized Hosting

Hosting test results in a central location with easy access via a browser enables the entire team to better participate in ensuring quality during the ongoing development process. Reports containing test results, timing, tracing logs, and built-in test documentation (supported in both C/C++ and Perl) all correlated together enhances the context for all team members to manage testing and optimize resolving failures.

Team Collaboration

Team collaboration on testing activities is also facilitated with auto-generated email notifications and messaging. Emails contain links to test failures where file names and line numbers are provided to aid in resolution. Messaging allows team members to more effectively communicate on the specifics of test results while minimizing information required in traditional emails, status reports, etc.

Retrieved from "https://www.stridewiki.com/index.php?title=What_is_Unique_About_STRIDE&oldid=14439"