Test execution¶
ecu.test provides for test execution, in addition to a clearly arranged execution window, a debugger and a check module which are described in more detail in the following section.
The behavior of trace analysis for each execution mode is documented in more detail in The Interface between test and trace analysis.
Execute package¶
Package and trace analysis execution can be started by means of the respective button on the button bar. By means of the black arrow next to the button you can choose to start only package execution or only trace analysis execution.
The dialog displayed during package execution is shown in the following screenshot:
The dialog consists of three main regions.
The upper region shows the current runtime of the execution and the active execution step. The execution consists of three main parts: Test case, Trace analysis and report creation.
During the execution of the test case, the left region shows an overview of the executed main package including the ratings of already executed steps. The active step is highlighted.
The right region shows the call stack. It consists of the currently executed test step and all sub packages passed to reach this test step.
The buttons at the lower left corner can be used to open the real-time log (Message window as separate window) and to deactivate the automatic scrolling in the package overview.
The current overall result of the execution and the button to abort the execution are positioned at the lower right corner.
Run test case and trace analysis¶
This button starts test case and trace analysis execution.
Run test case and trace analysis parameterized¶
ecu.test allows carrying out any test case with a certain assignment of variables. When the respective button has been activated a dialog opens in which all defined variables are laid out clearly. In that dialog the values of variables can be adapted to your ideas.
By clicking on OK you confirm the current assignment and so carry out the test case with the desired variable assignment.
Info
If no variables have been defined in a given package the parameterization dialog does not appear.
Run test case¶
This button starts test case execution. Analysis job of the trace analysis are created but not executed.
Run test case and run trace analysis downstream¶
See Project execution with downstream analysis job execution.
Run trace analysis from current test report¶
This option makes it possible to re-run the analyses from the current report.
Run trace analysis separately¶
This button starts test case and trace analysis execution. Since no test case is executed, only manually added recordings in the recording groups are analyzed.
Run package interactively¶
Via this button the package can be executed in interactive mode. This means that the blocks that are marked as User instruction will be shown in a larger window and the content of action, parameter and expectation will be read loudly from a computer voice. The execution of such blocks is done manually by the tester. More information can be found in chapter Interactive testing.
With the small black arrow next to the start symbol can be chosen between the two execution modes.
Run test case and trace analysis interactively¶
This button starts test case and trace analysis in interactive mode. The trace analysis offers no special interactive functionality, it will be executed automatically and without user interaction as usual.
Run test case and trace analysis interactively parameterized¶
This button combines the functionality of the following two:
Run test case interactively¶
This button starts the test case without trace analysis. The execution without trace analysis is recommended if the results of the trace analysis are not needed immediately and the runtime of the trace analysis shall be saved. The trace analysis can be executed later via menu entry Run trace analysis separately.
Debug package¶
A debugger is integrated in ecu.test for carrying out a simple analysis if problems arise.
The debugger is started by the button next to “normal” test execution. The arrow next to it allows the choice of carrying out only the test or also a trace analysis (if any).
When the debugger is started execution runs as with a normal start, but only up to the first breakpoint. From there the continued execution can be controlled by means of several options:
continue execution up to next breakpoint
debug reference (this means if the step is a package every single step has to be confirmed in the called package)
carry out next step only (and wait for next action).
Breakpoints are set or removed by the test step context menu (right click on a test step) and represented by a red dot on the test step (see figure).
Evaluate test report¶
The test report is made available in a database. A database view opens automatically for easy analysis. The test report presents all test steps with their evaluation (SUCCESS, FAILED, ERROR). The evaluation is displayed either by a background color in the table line (this test step has that evaluation) or just by a colored frame. The latter means the test step itself has no evaluation but further test steps are subordinated to it, and thus it is the evaluation of the subordinate test steps that is given ( e.g. in case of package references, block or If-Then-Else steps).
All functions concerning a test report are explained in detail in test report.
Test case coverage¶
With the test case coverage functionality, the line coverage of individual test steps can be measured during test case execution and then visualized in the package editor.
- Use case
Creation of coverage metrics in terms of quality management
Determination of branches of the test case that have not been entered for the purpose of cleanup
Protection of central user libraries (e.g. library workspaces)
- Activation
To measure the coverage during test execution, this must be activated in the test configuration:
- Handling
If the coverage measurement has been activated in the test configuration, it is determined directly when a package or project is executed. The line coverage is only visualized within packages.
The visualization is applied to all packages opened in the editor directly after the test case execution.
It is also possible to load additional coverage files and reset the visualization via the corresponding submenu of Extras.
Coverage files can also be loaded from test.guide via . To do this, a project filter must be created in test.guide. All coverage files contained in the filtered reports can then be loaded by specifying the project filter ID. To do this, a connection to test.guide must be configured. This can be done in the settings under . When creating a new test.guide instance, the server address, port and authentication key are particularly important.
To be able to store the coverage files on test.guide, the expression
Coverage Info/*.xmlmust be added to the test.guide instance under .- Generate a coverage report
Via , it is also possible to create a coverage report that summarizes all currently loaded coverage files. The following files are created:
An XML file containing the merged coverage data
An HTML file with a graphical representation of the test step coverage of all packages within a workspace
A JSON file with detailed coverage information for each package
The HTML representation also makes it possible to open packages in the test case editor and view their test step coverage in detail.
The schema definition of the JSON file is located in the installation directory under
res/schemas.