The Interface between test and trace analysis¶

Test and trace analysis are independent and can be executed separately.

The interface between both are the analysis jobs.

By means of analysis jobs it is possible to run trace analyses detached from the tests from which they originally stem. The analysis jobs can be processed at a later time, even on a separate computer (see Subsequent Execution of analysis jobs), or in parallel to the run of an ongoing test project (see Project execution).

Analysis jobs are stored in XML-files with the ending .ajob, which contains the following information:

• A reference to the trace analysis of the test package

• A reference to the test configuration that is needed for the analysis

• References to the traces that shall be analyzed

• A signal mapping and binding

• Optionally a synchronization declaration for the provided traces

• A set of parameter values

Certain events in the test case (especially the test steps Exit and Return) can prevent the execution of trace analyses (as well as the prior synchronization of traces) under certain circumstances. The following is an overview of the conditions under which the traces are not synchronized and the trace analysis is not performed.

Generation of analysis jobs during test runs¶

Analysis jobs are generated automatically during test runs and are stored within the report directory. Within the test report the generation of analysis jobs is documented with test step “Analysis job” at the end of a package.

You can read more on the specifics of analysis jobs generation here.

Test execution with analysis job execution¶

The entry “Run test case and trace analysis (F9)” of the Play button of the package editor runs the test case at first. Jobs that are generated within the test case are then executed automatically.

Test execution without analysis job execution¶

The entry “Run test case (Ctrl + F9)” of the Play button of the package editor only runs the test case. Analysis jobs are created during such a test run, but they are not executed automatically.

The created analysis jobs can be executed manually afterwards (see Subsequent Execution of analysis jobs).

Test execution with downstream analysis job execution¶

See Project execution with downstream analysis job execution.

Trace analysis execution without test execution¶

The entry “Run trace analysis only (Ctrl + F8)” of the Play button of the package editor runs the trace analysis for the traces that have been added manually under Signal recordings. This happens independently of the actual test case.

Project execution¶

Project execution with analysis job execution¶

The entry “Run project (F9)” of the Play button of the project editor runs the project. After the test execution of a package in the project, the created analysis jobs are executed automatically, before the next package proceed.

Project execution without analysis job execution¶

The entry “Run project without executing analysis jobs (Ctrl + F9)” of the Play button of the project editor only runs the test cases of the project. Analysis jobs are created within each package, but they will not be executed automatically.

The created analysis jobs can be executed manually afterwards (see Subsequent Execution of analysis jobs).

Project execution with parallel analysis job execution¶

The entry “Run project and execute analysis jobs in parallel (Ctrl + Shift + F9)” of the Play button of the project editor runs the project. Analysis jobs that are created within each package, will be executed one after another within a separate backround process in parallel to the ongoing project execution.

Project execution with downstream analysis job execution¶

For the downstream analysis the test case is executed locally and the analysis jobs are created. The latter are distributed by test.guide on configured analysis hosts for a more efficient execution. The following requirements are necessary for the downstream analysis:

1. test.guide has to be configured in ecu.test.

2. The workspace in which the project is executed has to be under version control.

3. test.guide has to be configured for the downstream analysis:

   • Under Test execution - Settings the downstream analysis must be configured.

     • The downstream analysis must be active.

     • The strategy for uploading artifacts must be chosen.

   • The SCM-repository has to be configured as SCM tool in test.guide.

   • An analysis host with a ResourceAdapter connected to test.guide is required. The ResourceAdapter has to be configured as XIL-type “Analysis” in the Configurations-Plugin.

Execute trace analyses only¶

The entry “Execute trace analyses only” of the Play button of the project editor only runs the analysis jobs of the packages in the project. Please notice that only the content of the trace analyses will be examined. Elements in the test case like the test step “Add Trace” or recording of test case variables are completely ignored.

Documentation of analysis jobs within the test report¶

Each analysis job that is executed receives a separate report. If an analysis job is executed automatically from a test run (either from a package or a project), that report that is created for the analysis job will be merged into the overall test report of the test execution from which the analysis job stems.

After a successful merge the individual analysis job report is deleted to keep project report directories clean. Automatic deletion of these reports can be disabled in the settings (see Settings for test execution).

A test report, whose test execution has generated analysis jobs, contains an entry for each generated job. In the report viewer each such entry is shown below the separate navigation tree entry “analysis jobs”. By clicking on such a job the report of the job is displayed on the right hand side of the report viewer. If the test and analysis reports have not been merged the analysis job is marked as “Inconclusive”.

If an analysis job was executed subsequently a post evaluation with a generated comment will be added to the report entry automatically. These post evaluation comments are used to keep track of multiple subsequent executions of an analysis job and its results.

Subsequent execution of analysis jobs¶

Manual execution of analysis jobs¶

The file ending .ajob is registered when installing ecu.test. A double-click on such an analysis job file within the Windows Explorer opens up ecu.test and brings up the analysis job dialog. The same job can also be opened directly in ecu.test by choosing File ‣ Open ‣ Analysis Job

The analysis job dialog lists all essential information from within the job and also provides a Play button to trigger the execution of this job. After execution the report viewer opens and shows the created analysis job report.

Execution of analysis jobs after execution of packages or projects out of the test report¶

The analysis jobs which are generated automatically during test runs can be executed subsequently out of the test report (as often as desired). It does not matter whether the analysis jobs have already been executed (see Test execution with analysis job execution, Project execution with analysis job execution or Project execution with parallel analysis job execution) or not (see Test execution without analysis job execution or Project execution without analysis job execution).

This subsequent execution of analysis jobs can be initiated as follows:

1. After execution of a package select “Run trace analysis from current test report (F8)” in the drop-down menu of the button Play in the package editor. The same function is also available in the test report by right-clicking on the main package as context menu entry Execute analysis jobs.

   After the execution of analysis jobs their results are integrated into the test report of the package. Previous evaluations are updated accordingly (see section Documentation of analysis jobs within the test report).

2. After execution of a project all analysis jobs can be executed by right-clicking on the main project and selecting Execute analysis jobs. An individual analysis job can be executed by right-clicking on it and selecting the context menu entry Execute analysis job.

   After the execution of analysis jobs their results are integrated into the test report of the project. Previous evaluations are updated accordingly (see section Documentation of analysis jobs within the test report).

3. Single analysis jobs can be executed separately (context menu entry Execute analysis job).

   In this case the result will not be integrated into the original test report. Instead a new report containing only the individual analysis job report will be opened.

   This is especially useful when reexecuting the same analysis again and again as it is common while developing the analysis.

Specifics on the creation of analysis jobs¶

Automatic creation of analysis jobs¶

The number of created analysis jobs depends on multiple factors:

• Number of defined trace analyses within the test package

• Number of traces that have been recorded within the recording groups

• Existence of stream based trace step templates within the trace analysis definition

For every trace analysis the following applies:

• No active stream-based trace step within the trace analysis means:

  • The number of created analysis jobs is exactly the number of recordings within the recording group that has the least number of recordings.

• At least one active stream-based trace step within the trace analysis means:

  • The number of created analysis jobs is exactly the number of recordings within the recording group that has the most number of recordings.

  • A job for which not all recording groups contain a trace will be executed in stream-based only mode, which means only the stream-based trace steps of the trace analysis are executed.

Traces in log recording groups

Log recording groups behave differently than signal recording groups when it comes to analysis job creation. All traces from a log recording group are passed to exactly the first trace analysis job that is created from the test case. In addition this only happens provided that the trace analysis contains any stream-based trace step, since only stream-based trace steps can handle log recordings.

Explicit creation of analysis jobs¶

As an alternative to the automatic creation of analysis jobs these may be created explicitly using the utility Enqueue analysis job. In comparison to the automatic mechanism this utility allows for more control and enables to realize conditional trace analyses.