Stride Overview: Difference between revisions

From STRIDE Wiki
Jump to navigation Jump to search
No edit summary
Line 134: Line 134:


= Test Implementation =
= Test Implementation =
The test validation can be implemented in both ''native code'' on the target and ''script'' on the host.  
Stride based tests are integrated with the existing build system enabling your software to be both fully ''functional'' and ''testable'' at the same time. The ''test logic'' is separated from the application source code and is '''not''' executed unless invoked via the Stride Runner. Any ''source instrumentation'' is only active when executing tests. The impact of built-in testability to the software application is nominal.


===Tests in C/C++===
The test validation can be implemented in both ''C/C++'' on the target and ''Perl'' on the host.  
Writing API/Unit tests in [[Test_Units_Overview | '''native code''']] is the simplest way to begin validating the software. There is no new language to learn, no proprietary editor, and your normal programming workflow is not interrupted. Also there is no special APIs required to register tests, suites, etc. Just write your test in any combination of C/C++ and the auto-generated [[Intercept_Module | '''intercept module''']] via the [[Build_Tools | '''STRIDE build tools''']] takes care of everything. Tests from separate teams are automatically aggregated by the system -- no coordination is required.  


This type of testing works well for:


* Calling APIs directly
'''C/C++ Test Unit Example'''
* Validating C++ classes
<source  lang=cpp>
* Isolating modules
#include <srtest.h>
* Critical processing / timing
 
* and much more...
class MyTest {
public:
    void CheckFoo();
    void CheckBoo();
};
#pragma scl_test_class(MyTest)
</source>


===Tests in Script===
STRIDE also supports writing tests in [[Test_Modules_Overview | '''Perl script''']]. When writing test scripts there are minimal dependencies on the software build process. Tests can also validate global functions, setup conditions from the host, etc.


Scripts are well suited for Integration Testing focusing on:
<source  lang=cpp>
#include <mytest.h>


* State Machines
void MyTest::CheckFoo() {
* Data flow through system components
    .. 
* Sequencing between threads
    srEXPECT_EQ(foo(2 + 2), 4);
* and much more...
}
void MyTest::CheckBoo() {
    ..
    srEXPECT_GT(boo(3 * 3), 7);
}
</source>


In either case, the validation is done without impacting the application's performance.


===Testable Builds===
'''Perl Test Script Example'''
Stride enables software builds to be both fully ''functional'' and ''testable'' at the same time. Built-in ''automation and reporting'' is similar in concept to a ''debug build'' except the focus is on testing.
<source lang="perl">
use strict;
use warnings;


The ''testable build'' leverages the existing software build process by integrating into the same ''make system'' used by the development team (no one-off or special builds). Automatically included in the build is test automation controllable from the host. When tests are executed, an xml is generated on the host (and optionally uploaded to Testspace) which includes detailed test results and timing analysis. This xml file uses a custom schema for representing the ''hierarchy of results'' (suites, cases, etc.). Also included is a stylsheet specification (which will be written to the same directory as the xml file) that allows the results to be viewed as HTML in a browser.
package MyTests;
use base qw(STRIDE::Test);
use STRIDE::Test;


Developers can easily pre-flight test their source code changes before ''committing'' them to a baseline. Builds can be ''automatically regression tested'' as part of the daily build process.
sub CheckFoo : Test
{
    EXPECT_EQ(Remote->foo(2 + 2), 4);
}


The ''testable software'' is still fully functional and works exactly the same as before. Whatever the software image was used for in the past -- system testing, developer debugging, etc. -- is still applicable. The Stride '''test logic''']] is separated from the application source code and is NOT executed unless invoked via the Stride Runner. Any ''source instrumentation'' is only active when executing tests. The impact of built-in testability to the software application is nominal.
sub CheckBoo : Test
 
{
The application can easily be switched back to a ''non-testable'' build by simply removing the ''STRIDE_ENABLED'' preprocessor directive and rebuilding. This flag controls all Stride related source code and macros; there are no changes required to the build process to enable or disable this functionality.
    EXPECT_GT(Remote->boo(3 * 3), 7);
 
}
 
===Fixturing===
Setting up your ''Software Under Test'' to be in the right state for a test is critical for repeatability. STRIDE supports a set of [http://en.wikipedia.org/wiki/Test_fixture '''Test fixture techniques'''] that include
* [[Test_Fixturing_in_C/C%2B%2B#Specifying___Fixturing_Methods | '''Setup/Teardown''']]
* [[Parameterized_Test_Units | '''Parameter passing to tests''']]
* [[File_Transfer_Services | '''Host/Target file transfer''']]
* [[Function_Capturing | '''Function remoting''']]
* etc.
 
These techniques encourage best-practices such as partitioning of fixturing code and actual test code, and--in addition--these same techniques can be leveraged to dynamically configure your tests at run-time.
 
 
===Other Features===
There are numerous other features that can be leveraged to facilitate deeper test coverage:
* Remoting global functions for script based API testing
* Built-in logging on test execution
* Dynamic test / suite creation
* and much [[Test_API | ''' more ... ''']]


1;
</source>


= Works with Testspace =
= Works with Testspace =

Revision as of 18:23, 3 July 2015

Stride™ is a test framework for writing C/C++ test cases executing on-device. Stride's unique host-target architecture allows easier and better On-Target/Device White-box testing. Software builds automatically become more testable; facilitating deeper code coverage opportunities.


Stride block diagram


Runner

Stride also contains a lightweight Runner application that is a host-based command-line utility for interactive and automated test execution. It also integrates seamlessly with Testspace for test content management.

Runtime

Stride includes a Runtime software source package that supports connectivity with the host system. It is integrated with embedded application software to enable testability to be compiled into the software with minimal impact on performance or the size of the application. The software is agnostic to the RTOS, CPU, and transport. It can be configured to support multi-processes, multi-threads, user/kernel spaces, or a simpler single application process. Typically the runtime is configured as a background thread and only executes when running tests.

Compiler

STRIDE 4.2 Framework-b.jpg

The Stride Compiler (aka Build Tools) are a set of command-line utilities that integrate with your target software build process. They preprocess standard C/C++ header files that contain STRIDE test declarations to auto-generate source code comprising a custom Intercept Module. The intercept module code is then built with the Stride Runtime to provide fixturing and harnessing test logic as part of your build image.

Cross Platform

Stride works on virtually any target platform. Stride's cross-platform framework facilitates a unified testing approach for all team members, enabling organizations to standardize on a test workflow that is independent of the target platform being used or what branch of software is being changed.

Runtime written in standard C

The Stride Runtime is written in standard C on top of a simple platform abstraction layer that enables it to work across platforms. It can be configured for a single process multi-threading environment or multiple process environments (i.e. Linux, Windows, Embedded RTOS, etc.). It is delivered as source code to be included in the application's build system. The transport between the host and target is configurable and supports Serial and TCP/IP by default. Custom transports are also available. The runtime has also been tailored specifically to embedded applications, overhead is minimal. It consumes very little memory for table and control block storage. Resource usage is configurable and can be tailored to the limitations of the target platform.

Integrates With Your Existing Build System

Stride also auto-generates intercept module used for harnessing and remote logic as source code during the make process, removing any dependencies on specific compilers and / or processors.

Supports Off-Target Testing

Testing can also be conducted using a standard desktop machine (Windows, Linux, or FreeBSD). The Stride SDKs allow for a seamless transition between the real target and an Off-Target host environment.


Rich set of Assertions

Test cases are typically constructed leveraging a large set of available assertion macros. On failures macros provide such details as file name, line number, and why the condition failed. Stride provides are large set of optional macros available for automatic report annotation in the case of failures.

#include <mytest.h>

void MyTest::CheckFoo() {
    ..  
    srEXPECT_EQ(foo(2 + 2), 4); 
}
void MyTest::CheckBoo() {
    ..
    srEXPECT_GT(boo(3 * 3), 7); 
}


Extensive Parameter Support

Parameterized testing is key to reducing test code duplication. Constructor arguments, name-value collections, and host file remoting are all supported. Stride provides an easy way for passing and accessing these types of parameters, allowing customization of test behavior at runtime.

Create an INI-type of file (i.e. myinput.ini)

Loopsize   = 10
InputFile  = ${MYPATH}/datacontent.bin
Toogle     = On

Pass the name-value collection using the Stride Runner

$ stride .. --run="MyTest(/path/to/myinput.ini)"

Now simply access your input within test logic using built-in services

{
  // .. doing stuff ..
  GetParam("Loopsize", buffer, size);
  //.. using parameters for test logic
}


Test Doubles & Test Points

Stride offers advanced testing techniques that enable deeper and more effective testing with less effort.

Test Doubles

For more advanced testing scenarios, dependencies can be doubled. This feature provides a means for intercepting C/C++ global functions on the target and substituting a stub, fake, or mock. The substitution is all controllable via the runtime, allowing the software to continue executing normally when not running a test.

For example something like the following could be useful:

void MySuite::test1()
{
  // inserting my own version of malloc()
  srDOUBLE_SET(malloc, my_malloc_stub);

  // calling a routine that uses malloc(). checking handled null correctly
  ret_code = FuncThatUsersMalloc(42);
  srEXPECT_EQ(ret_code, -1)

  // restoring the real malloc()
  srDOUBLE_RESET(malloc)
}


Test Points

Stride leverages source instrumentation to provide an additional validation technique called Expectation Testing that can be applied to the executing software application. The execution sequencing of the code, along with state data, can be automatically validated based on what is expected. This validation technique does not focus on calling functions / methods but rather verifies code sequencing. This can span threads, process boundaries, and even multiple targets, as the application(s) is running. Leveraging simple macros -- called Test Points -- developers strategically instrument the source under test.

/* a test point with formatted string payload */
srTEST_POINT_STR("IMPORTANT", "important date= %d", myVar);


The test validation is done without impacting the application's performance (the on-target test code is executed in a background thread and scripts are executed on the host machine). When failures do occur, context is provided with the file name and associated line number of the failed expectations. This type of validation can be applied to a wide-range of testing scenarios:

  • State Machines
  • Data flow through system components
  • Sequencing between threads
  • Drivers that don't return values
  • and much more ...

The following is a snippet of what the test code might look like

 srTestPointExpect_t expected[]={
    {"IMPORTANT"}, myCheckData,
    {"ANOTHER_TP"},
    {0}};

 // setup the expections
 srTestPointSetup(..);

 // start the operations

 srTestPointWait(handle, 1000);


Test Implementation

Stride based tests are integrated with the existing build system enabling your software to be both fully functional and testable at the same time. The test logic is separated from the application source code and is not executed unless invoked via the Stride Runner. Any source instrumentation is only active when executing tests. The impact of built-in testability to the software application is nominal.

The test validation can be implemented in both C/C++ on the target and Perl on the host.


C/C++ Test Unit Example

#include <srtest.h>
  
class MyTest {
public: 
    void CheckFoo();
    void CheckBoo();
};
#pragma scl_test_class(MyTest)


#include <mytest.h>

void MyTest::CheckFoo() {
    ..  
    srEXPECT_EQ(foo(2 + 2), 4); 
}
void MyTest::CheckBoo() {
    ..
    srEXPECT_GT(boo(3 * 3), 7); 
}


Perl Test Script Example

use strict;
use warnings;

package MyTests;
use base qw(STRIDE::Test);
use STRIDE::Test;

sub CheckFoo : Test
{
    EXPECT_EQ(Remote->foo(2 + 2), 4);
}

sub CheckBoo : Test
{
    EXPECT_GT(Remote->boo(3 * 3), 7);
}

1;

Works with Testspace

TBD



For more details refer to our FAQ