pytest - best design for fuzzing with limited parameters

user_3706707 0
Viewed 614

I was wondering about fuzzing in pytest and what is the best way to do that.
In the past I used hypothesis library to fuzz values, but it work best only when running each test many times.
Because my system is slow I want to be able to split the tests into 2 categories: "daily_run" and "regression":

  • daily_run will run each test 1 time
  • regression will run each test X times

Each run I want to be able to use random values. The problem is that the test parameters have "valid" range I want to stay in when fuzzing. For example:

@pytest.parametrize("month_number", [4])
def test_foo(month_number):
    # Test with that value

So in that example I get fixed value for the month number - 4. I'll give another example before explaining what I have tried:

@pytest.parametrize("month_number", [40])
def test_invalid_foo(month_number):
    # Test with that invalid value

So in the second example I test with an invalid value.
The range for the month number is obviously 1-12. I guess I can write some logic of getting a random value between 1-12 on valid month, and getting random value between -infinity - 0 & 13 - infinity. But that is a very verbose way to do that for month_number alone. On reality I got dozen of parameters and I don't want to have to write a functionality for each one.

Of course I can write some generic logic to do so, and use that generic logic on every parameter, but I still wonder if there is a better way.

Also don't forget the 2 categories - daily_run & regression.

What is the best practice to write fuzzed test with parameter limits?

fuzzing pytest python
1 Answers

for python fuzzing, take a look at the new atheris fuzzer. Like libfuzzer, it gives you a Dataprovider to feed inputs your target, which by default feeds bytes, but can also be configured to one of the following:

def ConsumeIntList(count: int, bytes: int)
def ConsumeIntListInRange(count: int, min: int, max: int)

Your categories of targets is confusing at best. Usually you fuzz a harness/target for a certain amount of time and potentially reuse the folder of testcases produced.
Or you have a set of harnesses/targets that you fuzz often and some not so often (but I doubt that would be a good idea).

user_65912780
Related