Is the Writer Monad Now Suitable for Large Lazy Lists?

There are two factors at play here. The way the <>/mappend calls are nested, and whether the entire log is kept in memory.

How are the <> calls nested?

This depends on how you write your code using Writer, not on Writer's implementation. To see why, lets cheat.

data Tree a = Nil | Leaf a | Node (Tree a) (Tree a)
  deriving (Show)

instance Semigroup (Tree a)
  where x <> y = Node x y

instance Monoid (Tree a)
  where mempty = Nil

This is not a proper monoid, since <> is not associative. x <> (y <> z) gives Node x (Node y z) whereas (x <> y) <> z) gives Node (Node x y) z. It allows us to tell after the fact whether the Writer's "log" was reduced left-nested or right nested.

go :: Int -> Writer (Tree Int) ()
go i
  | i < 5
    = do tell (Leaf i)
         go (i+1)
  | otherwise
    = pure ()

main :: IO ()
main = do
  let (result, log) = runWriter $ go 1
  putStrLn (render log)

render Nil = "Nil"
render (Leaf x) = show x
render (Node x y) = "(" ++ render x ++ ") <> (" ++ render y ++ ")"

With this, you get: (1) <> ((2) <> ((3) <> ((4) <> (Nil))))

Clearly right-nested. Hence how you could generate an infinite list as the "log" of a Writer and consume it as it was generated in a relatively small amount of space.

But swap the order of the tell and the recursion, so that it looks like this:

go :: Int -> Writer (Tree Int) ()
go i
  | i < 5
    = do go (i+1)
         tell (Leaf i)
  | otherwise
    = pure ()

And you get this: ((((Nil) <> (4)) <> (3)) <> (2)) <> (1). Now it's left-nested, infinite recursion does not work:

import Control.Monad.Writer

createInfiniteList :: Int -> Writer [Int] ()
createInfiniteList i = do
  createInfiniteList (i+1)
  tell [i]

main :: IO ()
main = do
  let x = execWriter $ createInfiniteList 1
  print x

This never prints anything, and consumes an ever-growing amount of memory.

Basically the structure of the <> calls is similar to the structure of your Writer expressions. Everywhere you bind in a call to another function (including the equivalent in a do-block), all of the <> calls resulting from that call will be "inside parentheses". So tell _ >> recurse results in right-nested <>s, while recurse >> tell _ results in left-nested <>s, and more complex call-graphs result in similarly-structured nesting of <>s.

Forcing the result builds the entire log

Another particular thing about your test program is that it doesn't use the "result" of the Writer at all, only the "log". Obviously if the recursion is infinite there's never any final result at all, but if we change your program like so:

import Control.Monad.Writer

createLargeList :: Int -> Writer [Int] ()
createLargeList i
  | i < 50000000
    = do tell [i]
         createLargeList (i+1)
  | otherwise
    = pure ()

main :: IO ()
main = do
  let (result, log) = runWriter $ createLargeList 1
  print $ length log
  print result

Then it behaves similarly; length consumes the list as it is produced and completes in short order (and relatively low memory usage). After that the () is readily available and is printed immediately.

But if we change it to print the result first:

import Control.Monad.Writer

createLargeList :: Int -> Writer [Int] ()
createLargeList i
  | i < 50000000
    = do tell [i]
         createLargeList (i+1)
  | otherwise
    = pure ()

main :: IO ()
main = do
  let (result, log) = runWriter $ createLargeList 1
  print result
  print $ length log

Then on my system this takes much longer, and consumes nearly 15 GB of RAM¹. It does have to materialise the log fully in RAM just to get at the final result, even when the <>s are right-nested and the log can be consumed lazily.

Technically I believe it is not building the list in memory, but rather a chain of thunks applying <> to singleton lists, which is just as long as the final list and probably uses more memory per link in the chain. The resulting list is still consumed by length as it is generated by forcing those thunks, but that doesn't really help since the entire thunk chain had to be generated to get at the final () result, instead of having the thunk chain itself generates as length demands more of the list.

¹ That's compiling like ghc foo.hs; if I compile with -O2 then it behaves similarly to printing the length of the log first. This is a fairly simple case for GHC to inline everything and figure out a better way of computing the same result; I wouldn't assume its optimisations could address this problem if the program were more complicated.