Is there a utility for easily writing parametrized tests in Haskell?

user_4122880 0
Viewed 91

On the one hand we have automated tests which check one specific example. On the other, we have property-based tests à la QuickCheck where we provide the property while the framework provides the examples, but we might have to explain how examples should be generated.

Somewhere between these two extremes lies the possibility of writing a single test or property and running it on multiple user-provided examples. What utilities exist in the Haskell testing landscape that help with writing such parametrized tests?

As a concrete example, here is how it might be done in Python's pytest. I want to check that the len function gives the correct output for a variety of inputs. This can be done by writing a single test that checks that the length of some input is what the tester expects it to be, and parametrizing the test with numerous examples of inputs and corresponding expected results.

from pytest import mark

param = mark.parametrize

@param('input, expected',
       ((''  ,    0),
        ('a' ,    1),
        ('b' ,    1),
        ('ab',    2),
        ('xx',    3), # deliberate mistake
        ('xyz',   3),
        ('aaabc', 5)
       ))
def test_len(input, expected):
    assert len(input) == expected

Which produces ouptut like this:

len_test.py::test_len[-0] PASSED                      [ 14%]
len_test.py::test_len[a-1] PASSED                     [ 28%]
len_test.py::test_len[b-1] PASSED                     [ 42%]
len_test.py::test_len[ab-2] PASSED                    [ 57%]
len_test.py::test_len[xx-3] FAILED                    [ 71%]  (appears red)
len_test.py::test_len[xyz-3] PASSED                   [ 85%]
len_test.py::test_len[aaabc-5] PASSED                 [100%]

========================= FAILURES ==========================
______________________ test_len[xx-3] _______________________
len_test.py:15: in test_len
    assert len(input) == expected
E   AssertionError: assert 2 == 3
E    +  where 2 = len('xx')
============ 1 failed, 6 passed in 0.04 seconds =============

Is there something similar in Haskell?

testing haskell
2 Answers

Here's how I do it with HUnit:

adjustToBusinessHoursReturnsCorrectResult :: [Test]
adjustToBusinessHoursReturnsCorrectResult = do
  (dt, expected) <-
    [
      (zt (2017, 10, 2) (6, 59,  4) 0, zt (2017, 10, 2) (9,  0,  0) 0),
      (zt (2017, 10, 2) (9, 42, 41) 0, zt (2017, 10, 2) (9, 42, 41) 0),
      (zt (2017, 10, 2) (19, 1, 32) 0, zt (2017, 10, 3) (9,  0,  0) 0)
    ]
  let actual = adjustToBusinessHours dt
  return $ ZT expected ~=? ZT actual

I simply use do notation with the list monad to create a list of tests ([Test]) that HUnit can execute.

Typically, I inline them, so that they look like this:

main :: IO ()
main = defaultMain $ hUnitTestToTests $ TestList [
  "adjustToBusinessHours returns correct result" ~: do
    (dt, expected) <-
      [
        (zt (2017, 10, 2) (6, 59,  4) 0, zt (2017, 10, 2) (9,  0,  0) 0),
        (zt (2017, 10, 2) (9, 42, 41) 0, zt (2017, 10, 2) (9, 42, 41) 0),
        (zt (2017, 10, 2) (19, 1, 32) 0, zt (2017, 10, 3) (9,  0,  0) 0)
      ]
    let actual = adjustToBusinessHours dt
    return $ ZT expected ~=? ZT actual

  ,
  "Composed adjust returns correct result" ~: do
    (dt, expected) <-
      [
        (zt (2017, 1, 31) ( 7, 45, 55)   2 , zt (2017, 2, 28) ( 7,  0,  0) 0),
        (zt (2017, 2,  6) (10,  3,  2)   1 , zt (2017, 3,  6) ( 9,  3,  2) 0),
        (zt (2017, 2,  9) ( 4, 20,  0)   0 , zt (2017, 3,  9) ( 9,  0,  0) 0),
        (zt (2017, 2, 12) (16,  2, 11)   0 , zt (2017, 3, 10) (16,  2, 11) 0),
        (zt (2017, 3, 14) (13, 48, 29) (-1), zt (2017, 4, 13) (14, 48, 29) 0)
      ]
    let adjustments =
          reverse [adjustToNextMonth, adjustToBusinessHours, adjustToDutchBankDay, adjustToUtc]
    let adjust = appEndo $ mconcat $ Endo <$> adjustments

    let actual = adjust dt

    return $ ZT expected ~=? ZT actual
  ]

I'm sure there's other ways to achieve that goal, but I like this one because it doesn't require any extra dependencies; it just leverages the capabilities of the language.

user_1260140

Remember that a QuickCheck property is (usually) simply a function. We can use QuickCheck to supply it with arguments, but we can just as easily supply them ourselves too.

prop_foo :: Foo -> Bar -> Bool

prop1 = prop_foo (Foo 6) (Bar 9)
prop2 = prop_foo (Foo 9) (Bar 2)
...

Also, QuickCheck allows to supply a custom generator, so you can control the random distribution of test data, but I'm pretty sure you can also use it to generate data in an entirely non-random way.

Then there's HUnit, where I'm pretty sure you could fmap a test subroutine over test data to generate a set of test cases. Indeed, I sometimes use this for testing parsers [which isn't terribly amenable to random testing]. I write a function that takes an input string and an expected parse tree, and then constructs a HUnit test case named for the input string, which runs the parser on that string and asserts the output matches the given expectation.

Really, I imagine there's a couple of ways of doing this.

(Are you aware of SmallCheck? It does essentially the same thing as QuickCheck, but generating inputs systematically rather than randomly. So it will try all possible inputs of a given size, and then move onto the next size up. Might not be appropriate for your use case, but worth knowing.)

user_10060100
Related