How do I reset Python runtime between pytest test_functions?

user_271118 0
Viewed 207

I am using pytest to test code that creates a prometheus Python client to export metrics.

Between the test functions the prometheus client does not reset (because it has an internal state) which screws up my tests.

I am looking for a way to basically get a new Pyhon runtime before all test function calls. That way the internal state of the prometheus client would hopefully reset to a state that it had when the Python runtime started to execute my tests.

I already tried importlib.reload() but that does not work.

prometheus pytest python
2 Answers

If you want to start each test with "clean" Prometheus client then I think best is to move it's creation and tear down to fixture with function scope (it's actually default scope), like this:

@pytest.fixture(scope=function)
def prometheus_client(arg1, arg2, etc...)
#create your client here
yield client
#remove your client here

Then you define your tests using this fixture:

def test_number_one(prometheus_client):
#test body

This way client is created from scratch in each test and deleted even if the test fails.

user_196152940

Straightforward approach: memorize current metric values before the test

First of all, let me show how I believe you should work with metrics in tests (and how I do it in my projects). Instead of doing resets, keep track of the metric values before the test starts. After the test finishes, collect the metrics again and analyze the diff between both values. Example of a django view test with a counter:

import copy
import prometheus_client
import pytest

from django.test import Client


def find_value(metrics, metric_name):
    return next((
        sample.value
        for metric in metrics
        for sample in metric.samples
        if sample.name == metric_name
    ), None)


@pytest.fixture
def metrics_before():
    yield copy.deepcopy(list(prometheus_client.REGISTRY.collect()))


@pytest.fixture
def client():
    return Client()


def test_counter_increases_by_one(client, metrics_before):
    # record the metric value before the HTTP client requests the view
    value_before = find_value(metrics_before, 'http_requests_total') or 0.0
    # do the request
    client.get('/my-view/')
    # collect the metric value again
    metrics_after = prometheus_client.REGISTRY.collect()
    value_after = find_value(metrics_after, 'http_requests_total')
    # the value should have increased by one
    assert value_after == value_before + 1.0

Now let's see what can be done with the registry itself. Note that this uses prometheus-client internals and is fragile by definition - use at your own risk!

Messing with prometheus-client internals: unregister all metrics

If you are sure your test code will invoke metrics registration from scratch, you can unregister all metrics from the registry before the test starts:

@pytest.fixture(autouse=True)
def clear_registry():
    collectors = tuple(prometheus_client.REGISTRY._collector_to_names.keys())
    for collector in collectors:
        prometheus_client.REGISTRY.unregister(collector)
    yield

Beware that this will only work if your test code invokes metrics registration again! Otherwise, you will effectively stop the metrics collection. For example, the built-in PlatformCollector will be gone until you explicitly register it again, e.g. by creating a new instance via prometheus_client.PlatformCollector().

Messing with prometheus-client internals: reset (almost) all metrics

You can also reset the values of registered metrics:

@pytest.fixture(autouse=True)
def reset_registry():
    collectors = tuple(prometheus_client.REGISTRY._collector_to_names.keys())
    for collector in collectors:
        try:
            collector._metrics.clear()
            collector._metric_init()
        except AttributeError:
            pass  # built-in collectors don't inherit from MetricsWrapperBase
    yield

This will reinstantiate the values and metrics of all counters/gauges/histograms etc. The test from above could now be written as

def test_counter_increases_by_one(client):
    # do the request
    client.get('/my-view/')
    # collect the metric value
    metrics_after = prometheus_client.REGISTRY.collect()
    value_after = find_value(metrics_after, 'http_requests_total')
    # the value should be one
    assert value_after == 1.0

Of course, this will not reset any metrics of built-in collectors, like PlatformCollector since it scrapes the values only once at instantiation, or ProcessCollector because it doesn't store any values at all, instead reading them from OS anew.

user_26502490
Related