Frequently Asked Questions About STRIDE: Difference between revisions
No edit summary |
No edit summary |
||
| Line 1: | Line 1: | ||
= | == Integration with STRIDE == | ||
== What | === What is the size of the STRIDE Runtime? === | ||
The [[STRIDE Runtime]] is tailored specifically to embedded applications, overhead is minimal. The runtime consume very little memory and can be configured to run at low priorities. Similarly, tracing overhead is minimized by collecting raw data on the target, and uploading tracing information at a low priority task to the host for processing. Resource usage is configurable and can be tailored to the limitations of the target platform. | |||
= | {| class="wikitable" | ||
|+ Typical resource usage | |||
! Aspect !! Resources | |||
|- | |||
| Runtime Code Space || TBD | |||
|- | |||
| Runtime Memory Usage || TBD | |||
|- | |||
| Threads || 2 Threads; configurable priority; blocked when inactive | |||
|} | |||
== | === What is the processing overhead? === | ||
Work-in-progress (WIP). | |||
== | == Source Instrumentation == | ||
=== What are the advantages of Test Points over logging? === | |||
Test Points can be used for direct validation during real-time execution. Logging systems, while often useful for historical record, can only be used for post-processing validation at best. What's more, since there is no standard way to post-process logs, users often rely on manual inspection or non-scalable homegrown solutions for validation. Test Point validation is fully automated using the STRIDE Framework which provides robust harnessing and reporting. Test Points also can include optional data payloads which can be very useful for validating state when a point is hit. Test Points have potentially lower impact on the system than standard logging since they '''(1)''' are only sent from the origin if a test is actively subscribed to test point (labels are used as a filter at runtime) and '''(2)''' test points can be completely disabled (no-opped) in your build by undefining a single preprocessor macro ('''STRIDE_ENABLED'''). | |||
=== | === What about source instrumentation bloat? === | ||
Any mature code base has some level of diagnostic bloat to it. This often takes the form of ad-hoc logging or debug statements. With STRIDE Test Points and Test Logs, you open your software to better automated test scenarios. All STRIDE instrumentation takes the form of single line macros, so the amount of instrumentation bloat is no worse than other typical ad-hoc diagnostics. What's more, the STRIDE macros are all designed to be quickly no-opped in a build via a single preprocessor macro, making it possible to completely eliminate any actual impact on certain builds, if so desired. | |||
=== Are all Test Points active? === | |||
No. Under normal testing scenarios, only the specific test points that are needed for a test are actually broadcast through the system. We accomplish this by setting test point filters (by label) on the system whenever one of the Test Point setup functions is called (in script or native code). These filters are reset or removed at the end of the test, so in general ''none'' of the test points are actually sent through the system if no test is currently active. That said, there are a few special use cases in which ''all'' test points become active in the system - namely, when tracing is activated on the host runner and when a specific test case uses a set that include ''ANYTHING'' or ''EVERYTHING_ELSE''. In general, however, the test points that are actually sent from the system are ''only'' those that are needed to execute the behavior validation for the current test. | |||
=== Will it affect performance? === | |||
Our experience on a wide-range of systems has shown minimal impact from the STRIDE instrumentation. The STRIDE Runtime has been designed to be small, portable, and readily configurable, allowing it to be optimized for the platform's specific characteristics. | |||
=== Should I leave Test Points in? === | |||
Yes. Once you have some behavior tests written, it's worthwhile to maintain that instrumentation and the corresponding tests, which allows you to run the tests on any stride-enabled build. All of the instrumentation macros are easily no-opped via a single preprocessor flag, so you can choose to effectively remove the instrumentation code on select builds (production/release builds, for example). The ultimate value of instrumentation is the continuous feedback you get by regularly executing the automated tests on the build. | |||
== Testing == | |||
=== What languages are supported? === | |||
Work-in-progress (WIP). | |||
=== | === Can I test using my host computer? === | ||
Work-in-progress (WIP). | |||
=== What testing methodologies are supported? === | |||
Work-in-progress (WIP). | |||
Test | == Test Automation == | ||
== | === What is continuous integration and why should I care? === | ||
The | The key principle of continuous integration is regular testing of your software--ideally done in an automated fashion. STRIDE tests are reusable and automated. Over time, these tests accumulate, providing more and more comprehensive coverage. By automating the execution of tests and results publication via Test Space with every software build, development teams gain immediate feedback on defects and the health of their software. By detecting and repairing defects immediately, the expense and time involved with correcting bugs is minimized. | ||
=== Does STRIDE support continuous integration? === | |||
Yes. The [[STRIDE Runner]] provides a straightforward means to connect to the device under test and execute the test cases you've implemented using the STRIDE Framework. The runner allows you to configure which tests to run and how to organize the results using subsuites in the report. Since the runner supports an option-based command line interface, this tool is easy to integrate with typical continuous integration servers. Please refer to [[Setting_up_your_CI_Environment|this article]] as well. | |||
The | |||
=== Where/how do you store test results? === | |||
When you execute your tests using the [[STRIDE Runner]], upon completion the results are written to an xml file. This xml file uses a custom schema for representing the hierarchy of results (suites, cases, etc.). These files also include a stylsheet specification (which will be written to the same directory as the xml file) that allows them to be viewed as HTML in a browser. You are free to store these files for future use/reference. The runner '''also''' supports direct uploading to [[STRIDE Test Space]] which is a hosted web application for viewing and collaborating on your test results. Once you are regularly executing tests, whether automatically or manually, we recommend you use the upload feature to persist and share your test results. | |||
== | === Can I get email containing test reports? === | ||
Yes. If you use [[STRIDE Test Space]] to store your results, you can optionally configure your test space(s) to automatically notify users when new results are uploaded. The email that is generated by Test Space contains only summary information and provides links so that you can view the complete report data. | |||
If you are using a continuous integration server to initiate your testing, it's likely that is supports different forms of notification when the testing is complete, so it's often possible to attach the xml report data as part of the CI server notification. | |||
== | == Getting Started == | ||
=== | === How long does it take to install STRIDE? === | ||
Work-in-progress (WIP). | |||
== | === What kind of training is required? === | ||
Work-in-progress (WIP). | |||
[[Category: Overview]] | [[Category: Overview]] | ||
Revision as of 15:16, 14 June 2010
Integration with STRIDE
What is the size of the STRIDE Runtime?
The STRIDE Runtime is tailored specifically to embedded applications, overhead is minimal. The runtime consume very little memory and can be configured to run at low priorities. Similarly, tracing overhead is minimized by collecting raw data on the target, and uploading tracing information at a low priority task to the host for processing. Resource usage is configurable and can be tailored to the limitations of the target platform.
| Aspect | Resources |
|---|---|
| Runtime Code Space | TBD |
| Runtime Memory Usage | TBD |
| Threads | 2 Threads; configurable priority; blocked when inactive |
What is the processing overhead?
Work-in-progress (WIP).
Source Instrumentation
What are the advantages of Test Points over logging?
Test Points can be used for direct validation during real-time execution. Logging systems, while often useful for historical record, can only be used for post-processing validation at best. What's more, since there is no standard way to post-process logs, users often rely on manual inspection or non-scalable homegrown solutions for validation. Test Point validation is fully automated using the STRIDE Framework which provides robust harnessing and reporting. Test Points also can include optional data payloads which can be very useful for validating state when a point is hit. Test Points have potentially lower impact on the system than standard logging since they (1) are only sent from the origin if a test is actively subscribed to test point (labels are used as a filter at runtime) and (2) test points can be completely disabled (no-opped) in your build by undefining a single preprocessor macro (STRIDE_ENABLED).
What about source instrumentation bloat?
Any mature code base has some level of diagnostic bloat to it. This often takes the form of ad-hoc logging or debug statements. With STRIDE Test Points and Test Logs, you open your software to better automated test scenarios. All STRIDE instrumentation takes the form of single line macros, so the amount of instrumentation bloat is no worse than other typical ad-hoc diagnostics. What's more, the STRIDE macros are all designed to be quickly no-opped in a build via a single preprocessor macro, making it possible to completely eliminate any actual impact on certain builds, if so desired.
Are all Test Points active?
No. Under normal testing scenarios, only the specific test points that are needed for a test are actually broadcast through the system. We accomplish this by setting test point filters (by label) on the system whenever one of the Test Point setup functions is called (in script or native code). These filters are reset or removed at the end of the test, so in general none of the test points are actually sent through the system if no test is currently active. That said, there are a few special use cases in which all test points become active in the system - namely, when tracing is activated on the host runner and when a specific test case uses a set that include ANYTHING or EVERYTHING_ELSE. In general, however, the test points that are actually sent from the system are only those that are needed to execute the behavior validation for the current test.
Will it affect performance?
Our experience on a wide-range of systems has shown minimal impact from the STRIDE instrumentation. The STRIDE Runtime has been designed to be small, portable, and readily configurable, allowing it to be optimized for the platform's specific characteristics.
Should I leave Test Points in?
Yes. Once you have some behavior tests written, it's worthwhile to maintain that instrumentation and the corresponding tests, which allows you to run the tests on any stride-enabled build. All of the instrumentation macros are easily no-opped via a single preprocessor flag, so you can choose to effectively remove the instrumentation code on select builds (production/release builds, for example). The ultimate value of instrumentation is the continuous feedback you get by regularly executing the automated tests on the build.
Testing
What languages are supported?
Work-in-progress (WIP).
Can I test using my host computer?
Work-in-progress (WIP).
What testing methodologies are supported?
Work-in-progress (WIP).
Test Automation
What is continuous integration and why should I care?
The key principle of continuous integration is regular testing of your software--ideally done in an automated fashion. STRIDE tests are reusable and automated. Over time, these tests accumulate, providing more and more comprehensive coverage. By automating the execution of tests and results publication via Test Space with every software build, development teams gain immediate feedback on defects and the health of their software. By detecting and repairing defects immediately, the expense and time involved with correcting bugs is minimized.
Does STRIDE support continuous integration?
Yes. The STRIDE Runner provides a straightforward means to connect to the device under test and execute the test cases you've implemented using the STRIDE Framework. The runner allows you to configure which tests to run and how to organize the results using subsuites in the report. Since the runner supports an option-based command line interface, this tool is easy to integrate with typical continuous integration servers. Please refer to this article as well.
Where/how do you store test results?
When you execute your tests using the STRIDE Runner, upon completion the results are written to an xml file. This xml file uses a custom schema for representing the hierarchy of results (suites, cases, etc.). These files also include a stylsheet specification (which will be written to the same directory as the xml file) that allows them to be viewed as HTML in a browser. You are free to store these files for future use/reference. The runner also supports direct uploading to STRIDE Test Space which is a hosted web application for viewing and collaborating on your test results. Once you are regularly executing tests, whether automatically or manually, we recommend you use the upload feature to persist and share your test results.
Can I get email containing test reports?
Yes. If you use STRIDE Test Space to store your results, you can optionally configure your test space(s) to automatically notify users when new results are uploaded. The email that is generated by Test Space contains only summary information and provides links so that you can view the complete report data.
If you are using a continuous integration server to initiate your testing, it's likely that is supports different forms of notification when the testing is complete, so it's often possible to attach the xml report data as part of the CI server notification.
Getting Started
How long does it take to install STRIDE?
Work-in-progress (WIP).
What kind of training is required?
Work-in-progress (WIP).