Instructions for using the process separation¶

Motivation¶

A mechanism to execute user-provided Python code (henceforth “user code”) in a separate process shall be provided for the following reasons:

• Improvement of the stability of ecu.test, because separate processes cannot influence each other (in case of programming errors, bugs in libraries, performance and memory troubles, etc.)

• Usage of libraries already present in ecu.test when a different version or special adaptations are needed, or when these strongly affect the runtime environment and/or other libraries

• Better modularization through proper separation of ecu.test code from user code. This approach also helps with the migration to Python 3.

• Using a different interpreter (e.g. 32 bit Python, Python 3)

• Proper parallelization with use of several CPU cores

User code is executed in many areas of ecu.test. Some of the existing extension points interlock deeply with ecu.test. Therefore, transparently and automatically making existing user code run in separate processes is not possible. Instead, there are these instructions that can be applied to adapt existing user code or develop new code in a way that enables it to be executed in its own process.

When should user code be moved to a separate process?¶

From a user’s point of view, this step should always be undertaken if:

• Large amounts of data are being kept in memory

• Alternate versions of python libraries are used, which are in conflict with the versions shipped with ecu.test

• A different/specific version of Python is needed

• Crashes through flaws in the used libraries or in user code itself are to be expected

• When using python extensions or C/C++ libraries, whose exception handling uses mechanisms that are not expected by ecu.test and can lead to program crashes due to unhandled errors

Architectural overview¶

What limitations must be considered?¶

Data in the respective other process can no longer be accessed “just like that”. Instead, explicit data transfer is necessary. Large amounts of data can be slow and/or memory intensive to transfer. Therefore, the separation between code parts executed “internally” and “externally” should be chosen in a way that keeps data exchange between the processes low (e.g. only parameterization, status and result, not extensive intermediate results).

Libraries and data types provided by ecu.test are, when a custom interpreter is used, not available in the external process executing the script. Therefore, for exchange of data between ecu.test and used code executed in a separate process, python base data types must be used.

The file ApiProxy.py shipped in this example workspace grants a separately executed process access to the Internal API of ecu.test. Moreover, the COM API and Object API can be used.

Conversion procedure¶

To make existing user code take advantage of the process separation, one can proceed as follows:

1. Identify and/or create separation lines:

   • Decide which code parts shall run in their own process and which in the ecu.test main process

   • Code that shall run in its own process needs to be a function or method (not a code section within a larger function/method)

   • parameters and return values of the functions should preferably be small objects and only use the following types:

     • base data types: int, float, str, unicode, NoneType

     • aggregations: list, set, tuple, dict

       • with arbitrary levels of nesting

     • no self-written classes and no ecu.test data types (Map, Curve, Matrix, ByteStream etc.)

       • replace by putting the relevant attributes into tuples or dictionaries

     • no library classes

       • if resorting to tuples or dictionaries is not possible for some reasons, the pickle module can be used to convert objects to strings and vice versa.

2. Add the UserProcess package to the PYTHONPATH

   • In the ecu.test settings under Python library, if there is not already a folder specified, configure an initially empty folder.

   • copy the folder UserProcess into the folder specified there

   • If needed, also copy the file ApiProxy.py

3. Move any program code that shall be executed in a separate process into its own *.py file

   • It must not be contained directly in a function variable, a UserPyModule, or a report, project/package/parameter generator, UserUtility or UserTool, but must be separate from it, in its own *.py file(s).

   • Where the program code was formerly called directly, create a ProcessSeparator instance and use CallAsynchronous or CallSynchronous (analogous to the examples in ProcessDataInOtherProcess.py or TwoSubProcessesWithFunctionVariable.pkg)

     • Depending on whether the separate process shall be executed in ecu.test’s internal python interpreter or in a custom one, pass a path to the desired python.exe to the ProcessSeparator constructor. Default is None for ecu.test’s internal interpreter.

       Info

       In order for an external python interpreter to be usable, its python environment needs to have the tblib library installed.

4. If the module(s) running in the separate process contains access to the Internal API, add the following python code:

   from ApiProxy import ApiProxy
   api = ApiProxy()
   

   After these two lines, api.* can be used normally, just like within ecu.test

   Access to the COM API and Object API requires no special steps. Other ecu.test APIs (e.g. the Report API) cannot be accessed from the separate process. Any access to these needs to happen in the code part that is not running in the separate process.

5. In case exceptions are expected in a function intended to be called asynchronously, it can optionally be extended as follows, so the errors are not silently swallowed:

   def myFunction(...):
       try:
           # original implementation of the function
   
       except Exception:
           import traceback
           traceback.print_exc()
   

   This causes any exception occurring within the function to be at least logged in ECU_TEST_ERR.log instead of being silently ignored. When calling the method synchronously, the exception gets passed to the caller, therefore this kind of handling is not needed.

Example¶

The example workspace contains packages and python code to demonstrate the usage.