Why would OutOfMemoryException be thrown while using PLINQ Take()?

After I upgraded to DotNet 4.5, a query started giving me OutOfMemoryExceptions.

The (distilled) query is:

var tests = new int[]{}
    .AsParallel()
    .GroupBy(_ => _)
    .Take(int.MaxValue)
    .ToArray();

I'm posting this for anyone with the same problem. I'll answer below.