How do I globally seed np.random.default_rng for unit tests

user_3344139 0
Viewed 760

The recommended way by numpy to create random numbers is to create a np.random.Generator like this

import numpy as np

def foo():
    # Some more complex logic here, this is the top level method that creates the rng
    rng = np.random.default_rng()
    return rng.random()

Now suppose I am writing tests for my code base, and I need to seed the rng for reproducible results.

Is it possible to tell numpy to use the same seed every time, regardless where default_rng() is called? This is basically the old behavior of np.random.seed(). The reason I need this is because I have many such tests and would have to mock the default_rng call to use a seed for each of them since in pytest you have to mock at the location where something is being used, not where it is defined. Thus mocking it globally like in this answer does not work.

With the old way, one could define a fixture that sets the seed for each test automatically inside conftest.py like this:

# conftest.py

import pytest
import numpy as np

@pytest.fixture(autouse=True)
def set_random_seed():
    # seeds any random state in the tests, regardless where is is defined
    np.random.seed(0)

# test_foo.py

def test_foo():
    assert np.isclose(foo(), 0.84123412)  # That's not the right number, just an example

With the new way of using default_rng, this seems to no longer be possible. Instead I would need to put a fixture like this in every test module that requires the rng to be seeded.

# inside test_foo.py, but also every other test file

import pytest
from unittest import mock
import numpy as np


@pytest.fixture()
def seed_default_rng():
    seeded_rng = np.random.default_rng(seed=0)
    with mock.patch("module.containing.foo.np.random.default_rng") as mocked:
        mocked.return_value = seeded_rng
        yield 

def test_foo(seed_default_rng):
    assert np.isclose(foo(), 0.84123412)
pytest random-seed numpy unit-testing python
2 Answers

The best I've come up with is to have a parametrizable fixture in the conftest.py like this

# conftest.py
import pytest
from unittest import mock
import numpy as np


@pytest.fixture
def seed_default_rng(request):
    seeded_rng = np.random.default_rng(seed=0)
    mock_location = request.node.get_closest_marker("rng_location").args[0]
    with mock.patch(f"{mock_location}.np.random.default_rng") as mocked:
        mocked.return_value = seeded_rng
        yield

This can then be used in each test like so:

# test_foo.py
import pytest
from module.containing.foo import foo

@pytest.mark.rng_location("module.containing.foo")
def test_foo(seed_default_rng):
    assert np.isclose(foo(), 0.84123412)  # just an example number

It's still not as convenient as before, but you only need to add the mark to each test instead of mocking the default_rng method.

user_33441390

If you want the full numpy API with a guarantee of stable random values across numpy versions, the short answer is - you can't.

You can use a workaround with the np.random.RandomState module, but you sacrifice the use of the current np.random module - there's no good, stable way around this.

Why numpy.random is not stable across versions

As of numpy v1.16, numpy.random.default_rng() constructs a new Generator with the default BitGenerator. But in the description of np.random.Generator, the following guidance is attached:

No Compatibility Guarantee

Generator does not provide a version compatibility guarantee. In particular, as better algorithms evolve the bit stream may change.

Therefore, using np.random.default_rng() will preserve random numbers for the same versions of numpy across platforms, but not across versions.

This has been true since the adoption of NEP 0019: Random number generator policy. See the abstract:

For the past decade, NumPy has had a strict backwards compatibility policy for the number stream of all of its random number distributions. Unlike other numerical components in numpy, which are usually allowed to return different when results when they are modified if they remain correct, we have obligated the random number distributions to always produce the exact same numbers in every version. The objective of our stream-compatibility guarantee was to provide exact reproducibility for simulations across numpy versions in order to promote reproducible research. However, this policy has made it very difficult to enhance any of the distributions with faster or more accurate algorithms. After a decade of experience and improvements in the surrounding ecosystem of scientific software, we believe that there are now better ways to achieve these objectives. We propose relaxing our strict stream-compatibility policy to remove the obstacles that are in the way of accepting contributions to our random number generation capabilities.

Workaround for testing with pytest

A section of the NEP is devoted to Supporting Unit Tests and discusses preserving guaranteed stream compatibility across versions and platforms in the legacy np.random.RandomState module. From the numpy docs on "Legacy Random Generation":

The RandomState provides access to legacy generators. This generator is considered frozen and will have no further improvements. It is guaranteed to produce the same values as the final point release of NumPy v1.16. These all depend on Box-Muller normals or inverse CDF exponentials or gammas. This class should only be used if it is essential to have randoms that are identical to what would have been produced by previous versions of NumPy.

The np.random.RandomState docs provide an example usage, which can be adapted for use with pytest. The important point is that functions making use of np.random.random and other methods must be monkeypatched using a RandomState instance:

Contents of mymod.py


import numpy as np

def myfunc():
    return np.random.random(size=3)

Contents of test_mymod.py

import pytest
import numpy as np
from numpy.random import RandomState

from mymod import myfunc

@pytest.fixture(autouse=True)
def mock_random(monkeypatch: pytest.MonkeyPatch):
    def stable_random(*args, **kwargs):
        rs = RandomState(12345)
        return rs.random(*args, **kwargs)
    
    monkeypatch.setattr('numpy.random.random', stable_random)

def test_myfunc():
    # this test will work across numpy versions
    known_result = np.array([0.929616, 0.316376, 0.183919])
    np.testing.assert_allclose(myfunc(), known_result, atol=1e-6)
user_38887190
Related