Test Integration: Difference between revisions

From STRIDE Wiki
Jump to navigation Jump to search
No edit summary
 
(33 intermediate revisions by 3 users not shown)
Line 1: Line 1:
In this article, we discuss establishment of continuous integration testing using STRIDE.
In this article, we discuss adding tests to your target build and running these tests.


The starting point of this article assumes that your have completed the steps described in the article [[Test Integration]].
The starting point of this article assumes that you have completed the steps described in the article [[Build Integration]].
 
== Creating TestSpace Containers ==
As a first step, we must create a project and test space into which our test results will be uploaded.
 
# Using your web browser, navigate to your company's testspace site and login<ref>Your testspace URL and login credentials are provided by S2 Technologies</ref>
# Click on the "All Projects" tab, then click the "Create New Test Space" button at the bottom of the All Projects page.
# Since we don't yet have a ''project'' created to act as a parent to the ''testspace'',
## Scroll to the bottom or the ''New s2_test_space'' page and click the ''New Project...'' link.
## In the ''add project'' box that appears, enter the Name as Stride Deployment, and Description as "sandbox for deployment tests".
## Once the fields are entered, click the Create button in the add project box.
# Back at the top of the New s2_test_space page, enter text as follows:
#* Name: TestIntro
#* Description: testintro sample tests
#* Max results to keep: 10
# If you would like to grant access to other users, check the appropriate boxes in the ''additional users''
# For now, skip the ''auto compare'' options, you can change these (and other options) later if desired
# In the Project list box, choose "Stride Deployment"
# Finally, click the "create" button
# On the subsequent page that is displayed, you will be able to manually upload results data, but leave the browser as it is, we will be automatically uploading results in a later step.


== Adding the TestIntro Tests to the Target Build ==
== Adding the TestIntro Tests to the Target Build ==
Here we will add the TestIntro sample tests to the target build. We suggest that you continue to include the STRIDE diagnostics in this build and all subsequent builds.
Here we will add the [[Test Intro Sample|TestIntro sample tests]] to the target build.<ref>The TestIntro sample is distributed in the [[Distribution Files#Samples|Samples package]].</ref> We will use the same technique used earlier in the [[Build_Integration#Building_the_Target_with_STRIDE_Diagnostics|Build Integration]] to incorporate these tests.
 
Copy the file <tt>s2_testintro_test.h</tt> to a designated SCL_DIRS directory (or alternative), include the TestIntro source files in the compile/link and perform a target build.


Download the binary to the target and start it running.
The [[Test Unit Pragmas|SCL pragmas]] that declare the TestIntro test units are in the file <tt>s2_testintro_test.h</tt>, so this file needs to be supplied as input to the STRIDE build tools. If you are using the Makefile technique described in [[Build_Integration#Build_Tool_Inputs|Build Integration]], you simply need to add <tt>[path]Samples/TestIntro/s2_testintro_test.h</tt> to the <tt>SCL_SRCS_H</tt> variable so that it will be picked up as input to <tt>s2scompile</tt>.  


== Running and Publishing the TestIntro Tests ==
In addition, you need to include the TestIntro source files in the target compile/link.
To execute the TestIntro tests, use a Windows or Linux host computer that has connectivity with the target system via your configured STRIDE transport (TCP/IP or seral). Additionally, make the generated STRIDE database (.sidb) visible to the host computer via a shared filesystem or a file copy to the host system.


If not already present, install the appropriate [[Package_Installation#Host_Tools | Host Tools]] package on the host computer.
Once the build is complete, download the binary to the target in preparation of running the tests.


We will now run [[Stride Runner|stride]] with options instructing it to upload results to your TestSpace site. The command line gets rather long when all this is specified, so it's recommended that you create an options file to submit the command line options.
== Running the TestIntro Tests ==
 
To execute the TestIntro tests, use a Windows, Linux or FreeBSD host computer that has connectivity with the target system via your configured STRIDE transport (TCP/IP or serial). Additionally, make the generated STRIDE database (.sidb) visible to the host computer via a shared filesystem or a file copy to the host system.
Create a text file named TestIntroOpts.txt and add the following text. You will need to fill in your particular argument values where brackets [ ] are shown.
 
<pre>
-r /(*)  
--database [path]/stride.sidb  
--device [arg]
-u
--testspace [https://mycompany.stridetestspace.com@user:pwd]
--project "Stride Deployment"
--space TestIntro
</pre>


stride -O TestIntroOpts.txt
If not already present, install the appropriate [[Desktop_Installation | desktop Framework package]] on the host computer.


To run the tests, ensure that your target system is running, then enter the following in a host console:
<source lang="bash">
stride --database=<path>/stride.sidb --device=<device_address> --run="*"
</source>


The text below should be output to your host console as the test progresses.
<pre>
<pre>
Loading database...
Loading database...
Connecting to device...
Connecting to device...
  runtime version: 4.1.01
Executing test units...
Executing test units...
   s2_testintro_cclass
   s2_testintro_cclass
Line 64: Line 35:
   s2_testintro_testpoints
   s2_testintro_testpoints
     > 3 passed, 0 failed, 0 in progress, 0 not in use.
     > 3 passed, 0 failed, 0 in progress, 0 not in use.
Summary: 9 passed, 2 failed, 0 in progress, 0 not in use.
  ---------------------------------------------------------------------
  Summary: 9 passed, 2 failed, 0 in progress, 0 not in use.
 
Disconnecting from device...
Disconnecting from device...
Saving result file...
Saving result file...
Uploading to test space...
</pre>
</pre>
== Viewing Results in TestSpace ==
== Next Steps ==
==Automated Regression Testing==


best to set things up so that you automatically
=== Uploading and Viewing Results in Test Space ===
* build,
Optionally, if you have Internet connectivity and access to your company's [[STRIDE Test Space|Test Space]]<ref>You need to have an account with access to a dedicated "Stride Deployment" project and "TestIntro" space created by your Test Space administrator</ref> you could run again the TestIntro tests and upload the results by executing:
* load binaries on the hardware,
* start target running,
* run Stride tests
* publish results to test space


== Organizing Tests ==
<source lang="bash">
put test results in suites
stride --database=<path>/stride.sidb --device=<device_address> -u --testspace=<user>:<pwd>@<mycompany>.stridetestspace.com --project="Stride Deployment" --space=TestIntro --run="*"
</source>


develop tests with namespace conventions
[[image:TestIntro_spacepage.jpg|thumb||TestIntro Test Space Page]]
Open your web browser, login to your company's test space site if needed and navigate to the '''All Projects''' page.


==Publication of Test Results==
You should now see the '''TestIntro''' test space listed under the '''Stride Deployment''' project, and see that TestIntro's '''Result Set Count''' is now 1.
Upload to test space


comparison with previously uploaded data
Click on the '''TestIntro''' link to be presented with the '''TestIntro''' space page. At the bottom of the page you will see a table displaying the uploaded result sets for this test space. Click on the '''Sequence_1''' link to show detailed results for this test run.


immediately see trends in
* number of tests in each suite and overall
*
== Best Practices ==
=== Use C++ Test Classes if Possible ===
If possible, use C++ Test Classes to write your test units. Test Classes offer extra features and are the easiest to use for programmers.
=== Use Namespacing to Organize Tests ===
Over time, you will end up with lots of test units. Grouping these tests for display or running will become important as the count of test units grows. (You can see an example of C namespacing in the TestIntro tests.)
<hr/>
== Notes ==
== Notes ==
<references/>
<references/>
[[Category:Deployment]]
 
[[Category:Installation]]

Latest revision as of 23:22, 25 September 2013

In this article, we discuss adding tests to your target build and running these tests.

The starting point of this article assumes that you have completed the steps described in the article Build Integration.

Adding the TestIntro Tests to the Target Build

Here we will add the TestIntro sample tests to the target build.[1] We will use the same technique used earlier in the Build Integration to incorporate these tests.

The SCL pragmas that declare the TestIntro test units are in the file s2_testintro_test.h, so this file needs to be supplied as input to the STRIDE build tools. If you are using the Makefile technique described in Build Integration, you simply need to add [path]Samples/TestIntro/s2_testintro_test.h to the SCL_SRCS_H variable so that it will be picked up as input to s2scompile.

In addition, you need to include the TestIntro source files in the target compile/link.

Once the build is complete, download the binary to the target in preparation of running the tests.

Running the TestIntro Tests

To execute the TestIntro tests, use a Windows, Linux or FreeBSD host computer that has connectivity with the target system via your configured STRIDE transport (TCP/IP or serial). Additionally, make the generated STRIDE database (.sidb) visible to the host computer via a shared filesystem or a file copy to the host system.

If not already present, install the appropriate desktop Framework package on the host computer.

To run the tests, ensure that your target system is running, then enter the following in a host console: 

stride --database=<path>/stride.sidb --device=<device_address> --run="*"

The text below should be output to your host console as the test progresses. 

Loading database...
Connecting to device...
Executing test units...
  s2_testintro_cclass
    > 1 passed, 1 failed, 0 in progress, 0 not in use.
  s2_testintro_flist
    > 2 passed, 1 failed, 0 in progress, 0 not in use.
  s2_testintro_testdoubles
    > 3 passed, 0 failed, 0 in progress, 0 not in use.
  s2_testintro_testpoints
    > 3 passed, 0 failed, 0 in progress, 0 not in use.
  ---------------------------------------------------------------------
  Summary: 9 passed, 2 failed, 0 in progress, 0 not in use.

Disconnecting from device...
Saving result file...

Uploading and Viewing Results in Test Space

Optionally, if you have Internet connectivity and access to your company's Test Space[2] you could run again the TestIntro tests and upload the results by executing: 

stride --database=<path>/stride.sidb --device=<device_address> -u --testspace=<user>:<pwd>@<mycompany>.stridetestspace.com --project="Stride Deployment" --space=TestIntro --run="*"
TestIntro Test Space Page

Open your web browser, login to your company's test space site if needed and navigate to the All Projects page.

You should now see the TestIntro test space listed under the Stride Deployment project, and see that TestIntro's Result Set Count is now 1.

Click on the TestIntro link to be presented with the TestIntro space page. At the bottom of the page you will see a table displaying the uploaded result sets for this test space. Click on the Sequence_1 link to show detailed results for this test run.

Notes

  1. ↑ The TestIntro sample is distributed in the Samples package.
  2. ↑ You need to have an account with access to a dedicated "Stride Deployment" project and "TestIntro" space created by your Test Space administrator
Retrieved from "https://www.stridewiki.com/index.php?title=Test_Integration&oldid=13987"