pytest Plugin¶

ecu.test code includes a pytest plugin that enables convenient reporting of ecu.test code test case results to test.guide. It automatically converts every assertion into a test step, collects parameters, user-defined properties, and captures logs, stdout or stderr.

Prerequisites¶

• ecu.test code (see Getting Started).

• Running test.guide instance

Setup¶

1. Set the required environment variables:

   • TESTGUIDE_URL: Base URL of your test.guide instance (without trailing slash).

   • TESTGUIDE_PROJECT_ID: Numeric project ID (tooltip when hovering over the project name).

   • TESTGUIDE_AUTH_KEY: Your personal authentication key (create it in test.guide under “Authentication keys”).

   The launch configuration of the provided demo workspace allows you to enter these values in a Visual Studio Code dialog but they can be set manually as well:

   Linux Bash

   export TESTGUIDE_URL="https://your.test-guide.instance"
   export TESTGUIDE_PROJECT_ID=42
   export TESTGUIDE_AUTH_KEY="xxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
   

   Windows CMD

   set TESTGUIDE_URL=https://your.test-guide.instance
   set TESTGUIDE_PROJECT_ID=42
   set TESTGUIDE_AUTH_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
   

1. Create (or extend) tests/pytest.ini to enable passed assertion reporting:

   [pytest]
   enable_assertion_pass_hook = True
   enable_verdict_none = true
   

   After changing this file, clear the Python bytecode (delete __pycache__ folders).

2. Add a minimal test (tests/test_quickstart.py):

   def test_add():
         assert 1 + 1 == 2
   

3. Run pytest and upload:

   pytest tests/test_quickstart.py --upload-to-testguide
   

4. A test.guide link with the appropriate filter should be visible in the Pytest report in the console. Review the test run in your test.guide project. If you enabled enable_assertion_pass_hook, the passed assertion appears as a test step with verdict PASSED.

Troubleshooting¶

• SSL issues: If you are using manually configured SSL certificates for your test.guide instance and you run into issues when uploading results to test.guide, you may want to install pip_system_certs to be able to use certificates configured on OS level. Alternatively you can configure pytest to skip certificate verification (not recommended). Further details can be found in the Configuration section.

• Test verdict is not PASSED: Confirm enable_assertion_pass_hook = True and remove stale .pyc files.

Configuration¶

Command Line¶

pytest command-line options are passed on the pytest command line and apply only to that single test run, e.g.:

pytest test_example.py --upload-to-testguide

The following table lists all command-line options provided by this pytest plugin. “Default” describes whether the feature is active if the option is not used.

pytest.ini¶

Entries in pytest.ini define persistent default values for plugin options (e.g. enable_assertion_pass_hook = true).

The following table lists all .ini options provided by this pytest plugin:

Test Outcomes and Verdicts¶

The pytest plugin represents each assertion as a test step. In order for passed assertions to also show up, the pytest option enable_assertion_pass_hook must be enabled. This is already preconfigured in the shipped pytest.ini. Make sure to clean your .pyc files after altering the enable_assertion_pass_hook configuration. You may also want to set enable_verdict_none in order to assign the verdict “NONE” to test cases without any assertions. Please note that any exceptions raised, other than AssertionError, will result in the test case being reported with the verdict ERROR in the test.guide report. Skipped tests will be reported with verdict INCONCLUSIVE. You can find a detailed overview of the verdicts and when they occur below.

Output Capture¶

This pytest plugin can include output captured by pytest via stdout, stderr, or logging as artifacts in the test.guide report. Additionally, whenever a test case has the verdict FAILED or ERROR, the pytest report is attached as an artifact to the test case.

Stdout and Stderr¶

This feature is enabled by default. As soon as content is captured on stdout or stderr within a test case, an artifact (stdout.log / stderr.log) containing the captured content is added to the test case. If no content is captured, no corresponding artifact is created. The feature can be disabled by turning off pytest capturing, for example using the CLI option -s.

pytest test_example.py -s --upload-to-testguide

For further information, have a look at Disabling Capturing in the pytest documentation.

Logging¶

Similar to stdout and stderr, this plugin can also add logs captured by pytest (from the logging library) as an artifact to a test case. If logs are captured within a test case, the artifact logs.log is added to the test case. For more information on enabling and configuring logging in pytest, see How to manage logging in the pytest documentation.

Pytest Report¶

As soon as a test case is assigned the verdict FAILED or ERROR, the pytest report is added to that test case as an artifact (<test_case_name>.log). The pytest report usually contains useful information such as the traceback, which helps to analyze errors in the test cases.

Property Recording¶

Pytest lets you attach arbitrary key/value metadata to a single test case via the record_property fixture (see pytest docs). These properties are reported in test.guide as Attributes of the corresponding test case. This feature is always active.

Example¶

def test_properties(record_property):
   record_property("ecu_sw_version", "v1.2.3")   # str value
   record_property("retries", 2)                 # int -> converted with repr()
   record_property("tags", ["smoke", "api"])     # list -> converted with repr()
   assert 1 + 1 == 2

Attributes in test.guide (all values as string):

Constants¶

Using the marker pytest.mark.constants(...), you can attach named constants to a test case. These values are reported in test.guide as Constants of the corresponding test case. The marker accepts keyword arguments only. It can be applied either as a decorator to a test function or test class, or by setting pytestmark at module level (see Example).

In addition, the constants fixture provides the constants set via pytest.mark.constants(...) as attributes for use within the test.

Example¶

import pytest

# Makes GLOBAL_CONSTANT available to all tests in this module
pytestmark = pytest.mark.constants(GLOBAL_CONSTANT="myValue")

# MAX_RETRIES is only available in test_constants
@pytest.mark.constants(MAX_RETRIES=3)
def test_constants(constants):
   for i in range(constants.MAX_RETRIES):
      do_something()

   # GLOBAL_CONSTANT is available via the pytestmark variable
   assert constants.GLOBAL_CONSTANT = "myValue"

# constants markers can be combined
# constants are reported even if the constants fixture is not used
@pytest.mark.constants(ARCH="x86", OS="linux")
@pytest.mark.constants(BUILD_TYPE="release")
def test_something():
   assert True

Argument Recording¶

Each test function’s arguments (including parametrized values and injected fixtures) are reported as test.guide parameters. This allows you to later filter or distinguish individual executions of a parameterized test case. For every test case argument, a test.guide parameter with direction IN is created. This feature is always active.

Example¶

import pytest

@pytest.mark.parametrize("x,y", [ (1, 2), (3, 4) ])
def test_params(x, y):
   assert (x + y) > 0

Result: two reported test.guide test cases (same name) with the following parameter values: