Parameterized Test Units: Difference between revisions

From STRIDE Wiki
Jump to navigation Jump to search
No edit summary
 
(7 intermediate revisions by 3 users not shown)
Line 1: Line 1:
You can leverage the tests you write in Test Units by using STRIDE's parameterization feature. This allows you to supply parameters to your test units at run time to customize test behavior.
__NOTOC__
Stride provides an easy way for passing parameters from the [[STRIDE Runner | Stride Runner]] on the host to the test code executing on the target device.  


==How Parameters are Passed from the Host==
Pass parameters via the command line to your constructor
Parameters to be passed to a target test unit are supplied on the stride.exe command line used to invoke the test unit. Test unit parameter values are specified between parentheses characters immediately following the test unit name on the command line.
 
  stride .. --run=MyTest(42)


===Number Parameter===
Pass parameters using a name-value collection
For example, assume that you have a test unit implemented on the target named MyTest and it has been coded to accept a single integer parameter. To run this test unit with the parameter value 17, use the following run specification:


<pre>
  stride .. --run="MyOtherTest(/path/to/file.ini)"
--run=MyTest(17)
 
</pre>


===String Parameter===
This allows customization of the test behavior at runtime and is useful when you want to run the same test scenario with different sets of input data, as described by [http://xunitpatterns.com/Parameterized%20Test.html this pattern].
As a second example, consider a test unit coded to take a single string as a parameter. The parameter is supplied as follows:


<pre>
There are two ways you could pass parameters:
--run=MyOtherTest(\"string\")
* as constructor arguments
</pre>
* as a name-value collection


Note that the string parameter must be enclosed with double quote characters. The double quotes are escaped with a backslash as this is required by the command line processor.
== Constructor Arguments ==
Passing parameters as constructor arguments is very simple and natural, however could be quite challenging when you need to pass more than a couple.  


====Parameter Containing Spaces====
=== How to use in your test code ===
 
To use parameters passed as constructor arguments you need to implement a public constructor in your C++ Test Unit (or extend the Initialization function of your C-Class Test Unit) with an argument list of [http://en.wikipedia.org/wiki/Null-terminated_string C-string] and numeric (integer or double) types:
When the string parameter contains one or more spaces, the command processor further requires you to enclose the entire <tt>--run</tt> argument in double quotes as follows:
 
<pre>
--run="MyOtherTest(\"this is the parameter\")"
</pre>
 
Note that the command line processor requires you to enclose with double quotes if there are any spaces at all in the <tt>--run</tt> argument. This can be counter-intuitive when passing only number types.
 
;No space in argument list
<pre>
--run=TestX(123,555,3.14)
</pre>
 
;Space in argument list
<pre>
--run="TestX(123, 555, 3.14)"
</pre>
 
===File Path Parameters===
File paths are a special case of string parameters; they require special handling due to their path delimiters.
 
On Windows systems, file paths are delimited using the backslash character ('\')
 
===Multiple Parameters===
You may pass any number of parameters to a test unit, and each parameter may be of any supported type. An example is shown below:
 
<pre>
--run="YetAnotherTest(19.58, -1, \"Encinitas\", 92024, \"south\")"
</pre>
 
A few additional examples are shown on the STRIDE Test Runner [[STRIDE_Runner#Test_Unit_Specification_Examples | Reference page]].
 
 
==How Parameters are Received on the Target==
Parameters are passed from the stride.exe command line and are presented on the target as follows:
 
; C++ test unit
: Parameters are presented as constructor arguments to the test unit class
 
; C-Class test unit
: Parameters are presented as arguments to the Initialization function.
 
; Flist test unit
: Not supported
 
===C++ Test Unit===
When using parameters with a C++ test unit, you declare and implement a public constructor that takes the number and type of arguments you want to receive.
 
Important: Do Not declare or implement a default constructor (constructor taking no arguments) in a class that gets parameters.
 
;MyTest.h


;C++ Test Unit
<source lang=cpp>
<source lang=cpp>
//MyTest.h
#include <srtest.h>
#include <srtest.h>


class MyTest : public stride:srTest
class MyTest
{
{
public;
public;
     MyTest(int nParam, const char* szParam)
     MyTest(int nParam, const char* szParam, double dParam)
        : m_nParam(nParam)
        , m_dParam(dParam)
     {
     {
        // typically, you will store parameter values into member variables for use by
         strncpy(m_szParam, szParam, MAX_VALUE_LEN);
        // the member test methods
        m_nParam = nParam;
         strncpy(m_szParam, szParam, MAX_LEN);
     }
     }


Line 94: Line 44:
     ...
     ...
private:
private:
    // typically, you will store parameter values into member variables for use by the member test methods
     int m_nParam;
     int m_nParam;
     char m_szParam[MAX_LEN];
     char m_szParam[MAX_VALUE_LEN];
    double m_dParam;
};
};


Line 102: Line 54:
#endif
#endif
</source>
</source>
'''NOTE:''' ''Don't declare more ''than one'' public constructor. This is typically not a problem for non-parameterized test classes, as a common [http://en.wikipedia.org/wiki/Nullary_constructor default constructor] is what's needed. Often you will begin with a non-parameterized test class and add the parameterization later as your testing gets more sophisticated. A common mistake here is to add a second public parameterized constructor to the class in addition to the original default constructor; this is perfectly legal in C++, but will not give you the desired result.''


===C-Class Test Unit===
'''NOTE:''' ''Don't set default values for your constructor arguments. In the case of Stride, each test class is instantiated dynamically on demand by the harnessing code. The harnessing code does not know at build time whether constructor parameters will be omitted when the class is ultimately instantiated. (See bellow how constructor arguments are specified on the Stride Runner command line on the host computer.)''
 
;C-Class Test Unit


;MyTest.h
<source lang=c>
<source lang=c>
/*MyTest.h*/
#include <srtest.h>
#include <srtest.h>
    
    
Line 127: Line 82:
</source>
</source>


;MyTest.c
 
<source lang=c>
<source lang=c>
/*MyTest.c*/
#include  "MyTest.h"
#include  "MyTest.h"
   
   
Line 139: Line 95:


/* other test implementations */
/* other test implementations */
...
...
...


void MyTest_Init(MyTest* self, int nParam, const char* szParam)
void MyTest_Init(MyTest* self, int nParam, const char* szParam)
Line 153: Line 105:
     ...
     ...
     ...
     ...
}</source>
}
</source>


==Supported Parameter Types==
=== How to pass via the Runner ===
To pass parameter values as constructor arguments you need to specify them between parentheses characters immediately following the Test Unit name on the [[STRIDE Runner | Stride Runner]] command line:


ANSI strings and number parameter types are directly supported.
;Number argument
Integer numbers could be specified as either decimal or hexadecimal. C++ <code>true/false</code> could also be specified.
--run=MyTest(17)
--run=MyTest(0x56)
--run=MyTest(21.95)
--run=MyTest(true)
;String argument
String arguments must be enclosed with double quotes. Further, the double quotes should be escaped with a backslash as this is required by the command line processor.
--run="MyOtherTest(\"this is the parameter\")"
--run="FileTest(\"c:\\path\\to\\file\")"
--run="FileTest(\"/path/to/file\")"
;Multiple arguments
--run="YetAnotherTest(19.58, -1, \"Encinitas, CA\", 92024, \"south\")"


{| class="wikitable"
'''NOTE:''' ''Be aware that the command line processor (console shell) imposes rules that requires any option's value containing special characters (e.g. space, ", ', |, \, /, *, ?...) to be enclosed within double-quotes. Within these quotes, all double-quote and backslash characters must be escaped with a preceding backslash.''
|-
| '''String types''' ||
* <tt>char*</tt>
* <tt>const char*</tt>
|-
| '''Number types''' ||
* ''all integer types''
* <tt>float</tt>
* <tt>double</tt>
|}


==Constructor Rules for Parameterized C++ Test Classes==
'''NOTE:''' ''You can specify only some of the argument values, as any omitted will default to empty for C-string or 0 for numeric.''
Because STRIDE test classes are instantiated dynamically by the STRIDE runtime, the rules for construction are subtly different from garden-variety C++ classes.


===Don't Declare More Than One Public Constructor===
== Name-Value Collection ==
You must declare '''only one''' public constructor in a test class.
Passing parameters as name-value collection is very powerful, however it requires explicit use of an API to obtain their values and as well a separate file on the host.  


This is typically not a problem for non-parameterized test classes, as a common default constructor <ref>more specifically a [http://en.wikipedia.org/wiki/Nullary_constructor nullary constructor] </ref> is what's needed. Often you will begin with a non-parameterized test class and add the parameterization later as your testing gets more sophisticated. A common mistake here is to add a second public parameterized constructor to the class in addition to the original default constructor; this is perfectly legal in C++, but will not give you the desired result.
=== How to use in your test code ===
 
To use parameters passed as name-value collection you need to call [[Runtime_Test_Services#class_srTest|srTest::GetParam]] for C++ (or [[Runtime_Test_Services#srTestGetParam|srTestGetParam]] for C).
You must ensure that you have only one public class constructor.
 
====Example====
<source lang="c" highlight="8">
// MyTest.h


;C++ Test Unit
<source lang=cpp>
//MyTest.h
#include <srtest.h>
#include <srtest.h>


class MyTest : public stride::srTest
class MyTest: public stride::srTest
{
{
public:
public;
     // Don't do this!
    MyTest()
    MyTest(); // WRONG! DON'T DECLARE DEFAULT (NULLARY) CONSTRUCTOR!
        : m_lParam(0)
        , m_dParam(0.0)
     {
        m_lParam = GetParam("name1", -1);
        m_dParam = GetParam("section.name1", 0.5772156649);
        GetParam("name2", m_szParam, MAX_VALUE_LEN, "default value");
    }


     // Do only this
     // Tests
     MyTest(int param1, const char* param2); // CORRECT: Declare only your paramterized constructor
     void Test1();
    ...
    ...
    ...
private:
    // typically, you will store parameter values into member variables for use by
    // the member test methods
    long m_lParam;
    double m_dParam;
    char m_szParam[MAX_VALUE_LEN];
};


  ...
#ifdef _SCL
};
#pragma scl_test_class(MyTest)
#endif
</source>
</source>


===Default Constructor Parameters Are Not Used===
Default constructor parameters are a very common and useful pattern in C++ programming. Where declared, default parameter values are inserted where appropriate by the C++ compiler at build-time (''static polymorphism'').


In the case of STRIDE, each test class is instantiated dynamically on demand by the harnessing code. The harnessing code does not know at build time whether constructor parameters will be omitted when the class is ultimately instantiated. (Recall that constructor parameters are specified on the stride.exe command line on the host computer.)
;C-Class Test Unit
 
Therefore, STRIDE imposes the following rules:
 
* Test class constructor arguments may be omitted
** Only trailing arguments may be omitted; they must be the last argument(s)
** The test class declaration does not need to specify default arguments for the above to work. (In fact the default arguments won't be used!)
 
* Any arguments that are omitted will be supplied to the constructor with the value 0 (or NULL for char pointer types).
 
====Example====
<source lang="c">
// MyTest.h


<source lang=c>
/*MyTest.h*/
#include <srtest.h>
#include <srtest.h>
 
typdef struct MyTest
{
    long m_lParam;
    char m_szParam[MAX_VALUE_LEN];


class MyTest : public stride::srTest
    void (*Test1)(struct MyTest* self);
{
     ...
public:
     ...
    // Default parameter values WILL NOT be used!
    // If a parameter is not specified on stride command line,
     // its value is always passed to the constructor as 0
     MyTest(int param1 = 100, int param2 = 200);  // WRONG! DON'T DO THIS!
     ...
     ...
};
} MyTest;
</source>


====A Pattern To Implement Non-Zero Default Construction Parameters====
void MyTest_Init(MyTest* self);


<source lang="c">
#ifdef _SCL
// MyTest.h
#pragma scl_test_cclass(MyTest, MyTest_Init)
#endif
</source>


#include <srtest.h>


#define MAX_STRING 256
<source lang=c>
 
/*MyTest.c*/
class MyTest : public stride::srTest
#include  "MyTest.h"
static void Test1(MyTest* self)
{
{
public:
    MyTest(int nParam, float fParam, const char* szParam);
     ...
     ...
    ...
    ...
}


private:
/* other test implementations */
    int m_nParam;
    float m_fParam;
    char m_szParam[MAX_STRING];
};


void MyTest_Init(MyTest* self)
{
    self->m_lParam = srTestGetParamLong("name1", -1);
    srTestGetParam("section.name2", self->m_szParam, MAX_VALUE_LEN, "default value");


// MyTest.cpp
    self->Test1 = Test1;
 
    ...
static const int nDEFAULT_NPARAM = 100;
    ...
static const float fDEFAULT_FPARAM = 3.1415;
    ...
static const char szDEFAULT_SZPARAM[] = "This is the default value";
}
 
</source>
MyTest::MyTest(int nParam, float fParam, const char* szParam)
{


    m_nParam = nDEFAULT_NPARAM;
=== How to pass via the Runner ===
    if (nParam)
To pass parameter values as name-value collection you need to create an [http://en.wikipedia.org/wiki/INI_file INI-formatted file] and specify it ('''without double quite''' encloser) between parentheses characters immediately following the Test Unit name on the [[STRIDE Runner | Stride Runner]] command line:
    {
--run="MyTest(/path/to/file.ini)"
        m_nParam = nParam;
    }


    m_fParam = fDEFAULT_FPARAM;
where "/path/to/file.ini" for example could be something like:
    if (fParam)
<source lang=ini>
    {
# line stating with a hash character are omitted
        m_fParam = nParam;
    }


    memset(m_szParam, 0, MAX_STRING);
# global values
    strncpy(m_szParam, szDEFAULT_SZPARAM, MAX_STRING - 1);
name1 = 17
    if (szParam[0])
name2 = this is a parameter
    {
complex.name1 = something else
        strncpy(m_szParam, szParam, MAX_STRING - 1);
    }
}


# section grouped valies
[section]
name1 = 13.86
name2 = yet another one
</source>
</source>


==Additional Tips When Using Parameters==
'''NOTE:''' ''The content of the INI-formatted file will be converted to a flat name-value collection, as any name "X" in section "Y" the file will appear under "Y.X" name in the collection. In case a name is repeated, only its last value will be present.''
 
===Setting the Test Unit Name Dynamically===
The purpose of parameterizing a test unit is to provide additional test coverage using a single test unit implementation. In a single stride.exe invocation you will typically run the same test unit many times, passing a different set of parameters for each test.
 
<small>
'''Note:''' Option files are a necessity, since the stride.exe command line gets unwieldy in these situations. Option files give you repeatibility and also eliminate a lot of typing!
</small>


===Test Units With Many Parameters===
'''NOTE:''' ''String values are not required to be double quote enclosed.''
----
;Notes
<references/>
[[Category: Test Units]]

Latest revision as of 15:36, 6 July 2015

Stride provides an easy way for passing parameters from the Stride Runner on the host to the test code executing on the target device.

Pass parameters via the command line to your constructor

 stride .. --run=MyTest(42)

Pass parameters using a name-value collection

 stride .. --run="MyOtherTest(/path/to/file.ini)"
 

This allows customization of the test behavior at runtime and is useful when you want to run the same test scenario with different sets of input data, as described by this pattern.

There are two ways you could pass parameters:

  • as constructor arguments
  • as a name-value collection

Constructor Arguments

Passing parameters as constructor arguments is very simple and natural, however could be quite challenging when you need to pass more than a couple.

How to use in your test code

To use parameters passed as constructor arguments you need to implement a public constructor in your C++ Test Unit (or extend the Initialization function of your C-Class Test Unit) with an argument list of C-string and numeric (integer or double) types:

C++ Test Unit
//MyTest.h
#include <srtest.h>

class MyTest
{
public;
    MyTest(int nParam, const char* szParam, double dParam)
        : m_nParam(nParam)
        , m_dParam(dParam)
    {
        strncpy(m_szParam, szParam, MAX_VALUE_LEN);
    }

    // Tests
    void Test1();
    ...
    ...
    ...
private:
    // typically, you will store parameter values into member variables for use by the member test methods
    int m_nParam;
    char m_szParam[MAX_VALUE_LEN];
    double m_dParam;
};

#ifdef _SCL
#pragma scl_test_class(MyTest)
#endif

NOTE: Don't declare more than one public constructor. This is typically not a problem for non-parameterized test classes, as a common default constructor is what's needed. Often you will begin with a non-parameterized test class and add the parameterization later as your testing gets more sophisticated. A common mistake here is to add a second public parameterized constructor to the class in addition to the original default constructor; this is perfectly legal in C++, but will not give you the desired result.

NOTE: Don't set default values for your constructor arguments. In the case of Stride, each test class is instantiated dynamically on demand by the harnessing code. The harnessing code does not know at build time whether constructor parameters will be omitted when the class is ultimately instantiated. (See bellow how constructor arguments are specified on the Stride Runner command line on the host computer.)

C-Class Test Unit
/*MyTest.h*/
#include <srtest.h>
  
typdef struct MyTest
{
    int m_nParam;
    char m_szParam[MAX_LEN];

    void (*Test1)(struct MyTest* self);
    ...
    ...
    ...
} MyTest;

void MyTest_Init(MyTest* self, int nParam, const char* szParam);

#ifdef _SCL
#pragma scl_test_cclass(MyTest, MyTest_Init)
#endif


/*MyTest.c*/
#include  "MyTest.h"
 
static void Test1(MyTest* self)
{
    ...
    ...
    ...
}

/* other test implementations */

void MyTest_Init(MyTest* self, int nParam, const char* szParam)
{
    self->m_nParam = nParam;
    strncpy(self->m_szParam, szParam, MAX_LEN);

    self->Test1 = Test1;
    ...
    ...
    ...
}

How to pass via the Runner

To pass parameter values as constructor arguments you need to specify them between parentheses characters immediately following the Test Unit name on the Stride Runner command line:

Number argument

Integer numbers could be specified as either decimal or hexadecimal. C++ true/false could also be specified.

--run=MyTest(17)
--run=MyTest(0x56)
--run=MyTest(21.95)
--run=MyTest(true)
String argument

String arguments must be enclosed with double quotes. Further, the double quotes should be escaped with a backslash as this is required by the command line processor.

--run="MyOtherTest(\"this is the parameter\")"
--run="FileTest(\"c:\\path\\to\\file\")"
--run="FileTest(\"/path/to/file\")"
Multiple arguments
--run="YetAnotherTest(19.58, -1, \"Encinitas, CA\", 92024, \"south\")"

NOTE: Be aware that the command line processor (console shell) imposes rules that requires any option's value containing special characters (e.g. space, ", ', |, \, /, *, ?...) to be enclosed within double-quotes. Within these quotes, all double-quote and backslash characters must be escaped with a preceding backslash.

NOTE: You can specify only some of the argument values, as any omitted will default to empty for C-string or 0 for numeric.

Name-Value Collection

Passing parameters as name-value collection is very powerful, however it requires explicit use of an API to obtain their values and as well a separate file on the host.

How to use in your test code

To use parameters passed as name-value collection you need to call srTest::GetParam for C++ (or srTestGetParam for C).

C++ Test Unit
//MyTest.h
#include <srtest.h>

class MyTest: public stride::srTest
{
public;
    MyTest()
        : m_lParam(0)
        , m_dParam(0.0)
    {
        m_lParam = GetParam("name1", -1);
        m_dParam = GetParam("section.name1", 0.5772156649);
        GetParam("name2", m_szParam, MAX_VALUE_LEN, "default value");
    }

    // Tests
    void Test1();
    ...
    ...
    ...
private:
    // typically, you will store parameter values into member variables for use by
    // the member test methods
    long m_lParam;
    double m_dParam;
    char m_szParam[MAX_VALUE_LEN];
};

#ifdef _SCL
#pragma scl_test_class(MyTest)
#endif


C-Class Test Unit
/*MyTest.h*/
#include <srtest.h>
  
typdef struct MyTest
{
    long m_lParam;
    char m_szParam[MAX_VALUE_LEN];

    void (*Test1)(struct MyTest* self);
    ...
    ...
    ...
} MyTest;

void MyTest_Init(MyTest* self);

#ifdef _SCL
#pragma scl_test_cclass(MyTest, MyTest_Init)
#endif


/*MyTest.c*/
#include  "MyTest.h"
 
static void Test1(MyTest* self)
{
    ...
    ...
    ...
}

/* other test implementations */

void MyTest_Init(MyTest* self)
{
    self->m_lParam = srTestGetParamLong("name1", -1);
    srTestGetParam("section.name2", self->m_szParam, MAX_VALUE_LEN, "default value");

    self->Test1 = Test1;
    ...
    ...
    ...
}

How to pass via the Runner

To pass parameter values as name-value collection you need to create an INI-formatted file and specify it (without double quite encloser) between parentheses characters immediately following the Test Unit name on the Stride Runner command line:

--run="MyTest(/path/to/file.ini)"

where "/path/to/file.ini" for example could be something like:

# line stating with a hash character are omitted

# global values
name1 = 17
name2 = this is a parameter
complex.name1 = something else

# section grouped valies
[section]
name1 = 13.86
name2 = yet another one

NOTE: The content of the INI-formatted file will be converted to a flat name-value collection, as any name "X" in section "Y" the file will appear under "Y.X" name in the collection. In case a name is repeated, only its last value will be present.

NOTE: String values are not required to be double quote enclosed.