I have a function that filters a few values out of a list of 1 billion integers. If I call it just once, memory use is nearly constant, but if I call it twice, it fills up over 8 gig of memory. This seems to only happen when I optimize with -O2 and when I use tuples to filter the list.
I worked down this testcase from code that did something more useful and was more complicated, but the function findLgMod3 demonstrates this problem, and the function findLgMod3Simp does not. These functions return the largest value <= 1 billion that is 3 mod the input. There are certainly better ways to do this, but they wouldn't demonstrate this problem.
I am compiling using ghc version 8.6.5 with:
ghc -O2 -dynamic map_mem_prob.hs
The code commented out shows variations that do and don't use excessive memory.
The code as written will use little memory when compiled unoptimized or with -O1, so I think it has to do with optimizing with -O2 and using tuples to filter the list, but I don't know if it is expected behavior and if so, how to re-work the function so that it still filters with tuples, but doesn't have this problem.
findLgMod3 :: Int -> Int
findLgMod3 modVal = maxAns
where
maxAns = if null allAns then 0 else last allAns
allAns = (map snd . filter (\(three, val) -> three == (val `rem` modVal))
. zip (repeat 3)) [1..1000000000]
findLgMod3Simp :: Int -> Int
findLgMod3Simp modVal = maxAns
where
maxAns = if null allAns then 0 else last allAns
allAns = (filter ((== 3) . (`rem` modVal))) [1..1000000000]
main :: IO ()
main = do
let val1 = 600000000
val2 = 700000000
let answers = map findLgMod3 [val1, val2] -- This blows up memory.
-- let answers = [findLgMod3 val1, findLgMod3 val2] -- This blows up memory.
-- let answer1 = findLgMod3 val1 -- This blows up memory.
-- answer2 = findLgMod3 val2
-- answers = answer1 + answer2
-- let answers = map findLgMod3Simp [val1, val2] -- This doesn't blow up memory.
-- let answers = map findLgMod3 [val1] -- This doesn't blow up memory.
-- let answers = [findLgMod3 val1] -- This doesn't blow up memory.
-- let answer1 = findLgMod3Simp val1 -- This doesn't blow up memory.
-- answer2 = findLgMod3Simp val2
-- answers = [answer1, answer2]
putStrLn $ "Answers: " ++ show answers
The output of the program should be:
Answers: [600000003,700000003]
However, it runs out of memory before it can complete.