Training Instrumentation - Revision history

Marku at 16:44, 29 August 2011

2011-08-29T16:44:09Z

← Older revision		Revision as of 16:44, 29 August 2011
Line 87:		Line 87:
	Now you've seen how instrumentation is strategically placed in source under test and can be traced during execution under the STRIDE Runner. We recommend that you proceed to some of our other training topics to learn how to create tests on the host (in script) or on the target (in native code) that use your instrumentation test points as a means for validation.		Now you've seen how instrumentation is strategically placed in source under test and can be traced during execution under the STRIDE Runner. We recommend that you proceed to some of our other training topics to learn how to create tests on the host (in script) or on the target (in native code) that use your instrumentation test points as a means for validation.

	[[Category: ~~Training~~]]		[[Category: Training_OLD]]

Ivailop: /* Build, Run, and Trace */

2011-06-09T19:14:06Z

Build, Run, and Trace

← Older revision		Revision as of 19:14, 9 June 2011
Line 70:		Line 70:

	<pre>		<pre>
	stride --device="TCP:localhost:8000" --database="../out/TestApp.sidb" --run=do_state_changes.pl --trace		stride --device="TCP:localhost:8000" --database="../out/TestApp.sidb" --run=do_state_changes.pl --trace=-
	</pre>		</pre>

Line 78:		Line 78:

	<pre>		<pre>
	stride --device="TCP:localhost:8000" --database="../out/TestApp.sidb" --run=do_state_changes.pl --trace --log_level=all		stride --device="TCP:localhost:8000" --database="../out/TestApp.sidb" --run=do_state_changes.pl --trace=- --log_level=all
	</pre>		</pre>

Ivailop: /* Build, Run, and Trace */

2010-06-10T19:41:45Z

Build, Run, and Trace

← Older revision		Revision as of 19:41, 10 June 2010
Line 58:		Line 58:
	So that you can see the trace points and logs in action, we will now build an off-target test app with this sample source included. Follow the [[Off-Target_Test_App#Copy Sample Source\|instructions here]], using the same source files from the Expectation sample that we just reviewed. Once the test app is built, we recommend that you manually open a new console (or Windows equivalent) and startup the application.		So that you can see the trace points and logs in action, we will now build an off-target test app with this sample source included. Follow the [[Off-Target_Test_App#Copy Sample Source\|instructions here]], using the same source files from the Expectation sample that we just reviewed. Once the test app is built, we recommend that you manually open a new console (or Windows equivalent) and startup the application.

	Now we want to [[Function_Capturing \| remotely invoke a function]] from our source under test using the [[STRIDE Runner]] and request that the runner show a trace of any trace points that occur in the test app during execution. Calling a function remotely in this example allows us to better control the behavior (state) of the software under test. In many cases you can simply trace on the application to view test points and test logs.		Now we want to [[Function_Capturing \| remotely invoke a function]] from our source under test using the [[STRIDE Runner]] and request that the runner show a trace of any trace points that occur in the test app during execution. Calling a function remotely in this example allows us to better control the behavior (state) of the software under test. In many cases you can simply [[Tracing\|trace]] on the application to view test points and test logs.

	In order to invoke the function that starts our state transitions, let's create the following two line perl script (use your favorite editor):		In order to invoke the function that starts our state transitions, let's create the following two line perl script (use your favorite editor):

Ivailop: /* What's The Difference between a Test Point and a Test Log ? */

2010-06-10T19:36:07Z

What's The Difference between a Test Point and a Test Log ?

← Older revision		Revision as of 19:36, 10 June 2010
Line 29:		Line 29:
	== What's The Difference between a Test Point and a Test Log ? ==		== What's The Difference between a Test Point and a Test Log ? ==

	[[Test Point\|Test Points]] can be used for validation since they are what's checked when you run a STRIDE expectation test. Test Logs, on the other hand, are purely informational and will be included in the report, according to the log level indicated when the [[Stride_Runner#Options\|STRIDE Runner]] was executed. Refer to the [[#Background\|background links above]] for more information on each of these instrumentation types.		[[Test Point\|Test Points]] can be used for validation since they are what's checked when you run a STRIDE expectation test. [[Test Log\|Test Logs]], on the other hand, are purely informational and will be included in the report, according to the log level indicated when the [[Stride_Runner#Options\|STRIDE Runner]] was executed. Refer to the [[#Background\|background links above]] for more information on each of these instrumentation types.

	== What About Data ? ==		== What About Data ? ==

Marku: /* Build, Run, and Trace */

2010-06-03T15:57:34Z

Build, Run, and Trace

← Older revision		Revision as of 15:57, 3 June 2010
Line 56:		Line 56:
	=== Build, Run, and Trace ===		=== Build, Run, and Trace ===

	So that you can see the trace points and logs in action, we will now build an off-target test app with this sample source included. Follow the [[Off-Target_Test_App#Copy Sample Source\|instructions here]], using the same source files from the Expectation sample that we just reviewed.		So that you can see the trace points and logs in action, we will now build an off-target test app with this sample source included. Follow the [[Off-Target_Test_App#Copy Sample Source\|instructions here]], using the same source files from the Expectation sample that we just reviewed. Once the test app is built, we recommend that you manually open a new console (or Windows equivalent) and startup the application.

	Now we want to [[Function_Capturing \| remotely invoke a function]] from our source under test using the [[STRIDE Runner]] and request that the runner show a trace of any trace points that occur in the test app during execution. Calling a function remotely in this example allows us to better control the behavior (state) of the software under test. In many cases you can simply trace on the application to view test points and test logs.		Now we want to [[Function_Capturing \| remotely invoke a function]] from our source under test using the [[STRIDE Runner]] and request that the runner show a trace of any trace points that occur in the test app during execution. Calling a function remotely in this example allows us to better control the behavior (state) of the software under test. In many cases you can simply trace on the application to view test points and test logs.

Marku: /* Build, Run, and Trace */

2010-06-03T15:50:20Z

Build, Run, and Trace

← Older revision		Revision as of 15:50, 3 June 2010
Line 58:		Line 58:
	So that you can see the trace points and logs in action, we will now build an off-target test app with this sample source included. Follow the [[Off-Target_Test_App#Copy Sample Source\|instructions here]], using the same source files from the Expectation sample that we just reviewed.		So that you can see the trace points and logs in action, we will now build an off-target test app with this sample source included. Follow the [[Off-Target_Test_App#Copy Sample Source\|instructions here]], using the same source files from the Expectation sample that we just reviewed.

	Now we want to invoke ~~the~~ function from our source under test using the [[STRIDE Runner]] and request that the runner show a trace of any trace points that occur in the test app during execution.		Now we want to [[Function_Capturing \| remotely invoke a function]] from our source under test using the [[STRIDE Runner]] and request that the runner show a trace of any trace points that occur in the test app during execution. Calling a function remotely in this example allows us to better control the behavior (state) of the software under test. In many cases you can simply trace on the application to view test points and test logs.

	In order to invoke the ~~[[Function_Capturing \|~~ function]] that starts our state transitions, let's create the following two line perl script (use your favorite editor):		In order to invoke the function that starts our state transitions, let's create the following two line perl script (use your favorite editor):

	<source lang="perl">		<source lang="perl">
Line 67:		Line 67:
	</source>		</source>

	Save this file as <tt>do_state_changes.pl</tt> in the <tt>~~src~~</tt> directory of your SDK. Now open a command prompt and change to that same directory. Execute the stride runner:		Save this file as <tt>do_state_changes.pl</tt> in the <tt>sample_src</tt> directory of your SDK. Now open a command prompt and change to that same directory. Execute the stride runner:

	<pre>		<pre>

Marku: /* Build, Run, and Trace */

2010-06-03T15:16:52Z

Build, Run, and Trace

← Older revision		Revision as of 15:16, 3 June 2010
Line 56:		Line 56:
	=== Build, Run, and Trace ===		=== Build, Run, and Trace ===

	So that you can see the trace points and logs in action, we will now build an off-target test app with this sample source included. Follow the [[Off-Target_Test_App#~~Build_Steps~~\|instructions here]], using the same source files from the Expectation sample that we just reviewed.		So that you can see the trace points and logs in action, we will now build an off-target test app with this sample source included. Follow the [[Off-Target_Test_App#Copy Sample Source\|instructions here]], using the same source files from the Expectation sample that we just reviewed.

	Now we want to invoke the function from our source under test using the [[STRIDE Runner]] and request that the runner show a trace of any trace points that occur in the test app during execution.		Now we want to invoke the function from our source under test using the [[STRIDE Runner]] and request that the runner show a trace of any trace points that occur in the test app during execution.

	In order to invoke the function that starts our state transitions, let's create the following two line perl script (use your favorite editor):		In order to invoke the [[Function_Capturing \| function]] that starts our state transitions, let's create the following two line perl script (use your favorite editor):

	<source lang="perl">		<source lang="perl">

Marku: /* Samples/test_in_script/Expectations/s2_expectations_source.c */

2010-06-03T15:05:01Z

Samples/test_in_script/Expectations/s2_expectations_source.c

← Older revision		Revision as of 15:05, 3 June 2010
Line 48:		Line 48:

	* ''SetNewState()'': This is a shared function that is called to effectuate a state change. The test point label is just a static string chosen by the instrumenter and the name of the new state is included as string data on the test point.		* ''SetNewState()'': This is a shared function that is called to effectuate a state change. The test point label is just a static string chosen by the instrumenter and the name of the new state is included as string data on the test point.
	* ''Start()'': The Start state includes a single test point with no data. The label is ~~gotten~~ from a ~~shard~~ function that returns a string representation for any given state (''GetStateName()'').		* ''Start()'': The Start state includes a single test point with no data. The label is from a shared function that returns a string representation for any given state (''GetStateName()'').
	* ''Idle()'': The Idle state records a single test point. The test point label is again obtained from ''GetStateName()'' and the data associated with the test point is a transition count value that the software under test is maintaining. This state function also includes an ''info'' level test log. Since it is an info level log, it will not be captured during testing unless you explicitly set the log level to ''info'' or higher when executing the tests.		* ''Idle()'': The Idle state records a single test point. The test point label is again obtained from ''GetStateName()'' and the data associated with the test point is a transition count value that the software under test is maintaining. This state function also includes an ''info'' level test log. Since it is an info level log, it will not be captured during testing unless you explicitly set the log level to ''info'' or higher when executing the tests.
	* ''Active()'': The Active state has three distinct test points. The first point is similar to previous states and records a test point with the name of the current state as a label and transition count as data. The second test point shows another example of including string data in a test point. The third test point shows an example of simple JSON serialization to include several values in a single test point payload.		* ''Active()'': The Active state has three distinct test points. The first point is similar to previous states and records a test point with the name of the current state as a label and transition count as data. The second test point shows another example of including string data in a test point. The third test point shows an example of simple JSON serialization to include several values in a single test point payload.

Mikee: moved ++WIP++ Training Instrumentation to Training Instrumentation

2010-06-01T17:08:34Z

moved ++WIP++ Training Instrumentation to Training Instrumentation

← Older revision	Revision as of 17:08, 1 June 2010
(No difference)

Mikee at 17:07, 1 June 2010

2010-06-01T17:07:28Z

← Older revision		Revision as of 17:07, 1 June 2010
Line 36:		Line 36:

	* Try to use string data payloads wherever possible. String payloads are considerably more human-friendly when viewing test results and they allow for relatively simple string comparison validation.		* Try to use string data payloads wherever possible. String payloads are considerably more human-friendly when viewing test results and they allow for relatively simple string comparison validation.
	* If you need to do complex validation of '''multi-field data''' in a test point, consider using an object serialization format such as [http://json.org/ JSON]. Standard formats like this are readily parsable in host scripting languages. If, however, you will only be writing expectation tests in native code for execution on the target, then string serialization formats might be too cumbersome for validation. In that case, using binary payloads (structures, typically) is reasonable.		* If you need to do complex validation of '''multi-field data''' in a test point, consider using an object serialization format such as [http://json.org/ JSON]. Standard formats like this are readily parsable in host scripting languages. If, however, you will ''only'' be writing expectation tests in native code for execution on the target, then string serialization formats might be too cumbersome for validation. In that case, using binary payloads (structures, typically) is reasonable.
	* ~~TBD~~		* The data payloads for a test point are limited to a fixed size (512 bytes by default, but configurable if needed). If you have large data payloads that you need to validate and you are using host-based script validation logic, consider using [[File Transfer Services]] to read/write the data as files on the host. If, on the other hand, you are using native code for validation, you can independently manage your own buffers of data (heap allocated, for example) for validation and use the test point payloads only to transmit addresses and sizes of the payloads.
	* TBD

	== Sample Code: s2_expectations_source ==		== Sample Code: s2_expectations_source ==
Line 46:		Line 45:
	=== Samples/test_in_script/Expectations/s2_expectations_source.c ===		=== Samples/test_in_script/Expectations/s2_expectations_source.c ===

	This is the source under test for a simple example that demonstrates expectation where the test logic is written in a script module (perl) that runs on the host. The source under test is a simple state machine and we have chosen to instrument each of the 4 states with one or more test points. Open the source file in your preferred editor and search for '''srTEST_POINT'''. You will see that we have test points in the following functions:		This is the source under test for a simple example that demonstrates expectation tests where the test logic is written in a script module (perl) that runs on the host. The source under test is a simple state machine and we have chosen to instrument each of the 4 states with one or more test points. Open the source file in your preferred editor and search for '''srTEST_POINT'''. You will see that we have test points in the following functions:

	* ''SetNewState()'': This is a shared function that is called to effectuate a state change. The test point label is just a static string chosen by the instrumenter and the name of the new state is included as string data on the test point.		* ''SetNewState()'': This is a shared function that is called to effectuate a state change. The test point label is just a static string chosen by the instrumenter and the name of the new state is included as string data on the test point.
	* ''Start()'': The Start state includes a single test point with no data. The label is gotten from a shard function that returns a string representation for any given state (''GetStateName()'').		* ''Start()'': The Start state includes a single test point with no data. The label is gotten from a shard function that returns a string representation for any given state (''GetStateName()'').
	* ''Idle()'': The Idle state records a single test point. The test point label is again obtained from ''GetStateName()'' and the data associated with the test point is a transition count value that the software under test is maintaining. This state function also includes an ''info'' level test log. Since it is an info level log, it will not be captured during testing unless you explicitly set the log level to ''info'' or higher when executing the tests ~~(via the STRIDE Runner)~~.		* ''Idle()'': The Idle state records a single test point. The test point label is again obtained from ''GetStateName()'' and the data associated with the test point is a transition count value that the software under test is maintaining. This state function also includes an ''info'' level test log. Since it is an info level log, it will not be captured during testing unless you explicitly set the log level to ''info'' or higher when executing the tests.
	* ''Active()'': The Active state has three distinct test points. The first point is similar to previous states and records a test point with the name of the current state as a label and transition count as data. The second test point shows another example of including string data in a test point. The third test point shows an example of simple JSON serialization to include several values in a single test point payload.		* ''Active()'': The Active state has three distinct test points. The first point is similar to previous states and records a test point with the name of the current state as a label and transition count as data. The second test point shows another example of including string data in a test point. The third test point shows an example of simple JSON serialization to include several values in a single test point payload.
	* ''Exp_DoStateChanges()'': This function drives the state transitions and includes one warning log message that is not hit during normal execution of the code.		* ''Exp_DoStateChanges()'': This function drives the state transitions and includes one warning log message that is not hit during normal execution of the code.
Line 86:		Line 85:
	== What next ? ==		== What next ? ==

	Now you've seen how instrumentation is strategically placed in source under test and can be traced during execution under the STRIDE Runner. We recommend you proceed to some of our other training topics to learn how to create tests on the host (in script) or on the target (in native code) that use your instrumentation test points as a means for validation.		Now you've seen how instrumentation is strategically placed in source under test and can be traced during execution under the STRIDE Runner. We recommend that you proceed to some of our other training topics to learn how to create tests on the host (in script) or on the target (in native code) that use your instrumentation test points as a means for validation.

	[[Category: Training]]		[[Category: Training]]