Positive numbers

You are on the right track in the sense that you look for the highest numbers. The problem is however that a, b and c are not always ordered.

Indeed say for instance that we have the numbers [6,2,4]. Then the way (a,b,c) will evolve through the recursion is:

(0,0,0) -> (6,0,0) -> (2,6,0) -> (4,2,6)

But now a=4, so that means that if we now encounter 3, we will not replace that value, whereas we can do this since we can remove the 2.

Although there are many ways to solve this, probably the best way to do this is to maintain order: ensure that a <= b <= c.

So we can use:

sol1 = sol2' (0,0,0)

sol2' (a,b,c) []     = a*b*c
sol2' t@(a,b,c) (x:xs) = sol2' f xs
  where f | x >= c = (b,c,x)
          | x >= b = (b,x,c)
          | x > a = (x,b,c)
          | otherwise = t

this produces the expected:

Prelude> sol1 [1,2,3,4]
24
Prelude> sol1 [3, 5, 1, 2, 4, 10, 0, 4, 8, 11]
880

Intermezzo: keep track of numbers if negative are present

Your program first takes (0,0,0) as the first three values. But in case for instance the list contains only negative numbers (i.e. [-1,-2,-3]) we of course want to keep track of these first. We can do this for instance by initializing our tuple with elements from the list:

import Data.List(sort)

sol1 (xa:xb:xc:xs) = sol2' (a,b,c) xs
    where [a,b,c] = sort [xa,xb,xc]

So now we take the first three elements, sort these, and use these as the first tuple. The remaining of the list is processed. This function will error in case sol1 is not given a list with at least three elements, but in that case there probably is no answer. We can use a Maybe to handle the fact that the function is non-total.

all numbers

Of course we also want to deal with negative numbers. Multiplying two negative numbers results in a positive number. So by keeping track of the two smallest numbers as well, we can then do the math properly. So first we will use another argument (d,e) to keep track of the smallest numbers with d <= e:

sol1_all = sol2_all' (0,0,0) (0,0)

sol2_all' (a,b,c) (d,e) []     = -- ...
sol2_all' t@(a,b,c) u@(d,e) (x:xs) = sol2_all' f g xs
  where f | x >= c = (b,c,x)
          | x >= b = (b,x,c)
          | x > a = (x,b,c)
          | otherwise = t
        g | x <= d = (x,d)
          | x <= e = (d,x)
          | otherwise = u

So now we have obtained the greatest numbers (a,b,c) and the smallest numbers (d,e). If d and e are indeed negative, then the only way to produce a large . So now we have the following possibilities to consider a*b*c and c*d*e. So we can write it as:

sol2_all' (a,b,c) (d,e) [] = max (a*b*c) (c*d*e)
sol2_all' t@(a,b,c) u@(d,e) (x:xs) = sol2_all' f g xs
  where f | x >= c = (b,c,x)
          | x >= b = (b,x,c)
          | x > a = (x,b,c)
          | otherwise = t
        g | x <= d = (x,d)
          | x <= e = (d,x)
          | otherwise = u

Note however that this will not always produce the correct result here because we can count two numbers in both tuples. We can solve this by properly initializing the tuples:

import Data.List(sort)

sol1_all (xa:xb:xc:xs) = sol2_all' (a,b,c) (a,b) xs
    where [a,b,c] = sort [xa,xb,xc]

sol2_all' (a,b,c) (d,e) [] = max (a*b*c) (c*d*e)
sol2_all' t@(a,b,c) u@(d,e) (x:xs) = sol2_all' f g xs
  where f | x >= c = (b,c,x)
          | x >= b = (b,x,c)
          | x > a = (x,b,c)
          | otherwise = t
        g | x <= d = (x,d)
          | x <= e = (d,x)
          | otherwise = u

Rationale behind picking different (possibly equivalent) elements

How do we know that we will not use an element twice? Since we only use a*b*c or c*d*e this will - in the case of a list with three element - boils down to max(a*b*c,a*b*c) (a, b, and c here the result of sort). So uniqueness is guaranteed. Since we will only add elements in the first tuple if these are at least greater than a, and less than b, we know that in order for an x to be added in both tuples, it should be a <= x <= b. In that case we will obtain tuples (x,b,c) and (a,x). But since we evaluate in that case x*b*c and a*x*c, x will thus not occur in any expression twice.

Leetcode challenge

I submitted a Python version of this code to the Leetcode Challenge and it was accepted:

class Solution:
    def maximumProduct(self, nums):
        a,b,c = d,e,_ = sorted(nums[:3])
        for x in nums[3:]:
            if x >= c:
                a,b,c = b,c,x
            elif x >= b:
                a,b = b,x
            elif x >= a:
                a = x
            if x <= d:
                d,e = x,d
            elif x < e:
                e = x
        return max(a*b*c,c*d*e)

There are somewhat more efficient solutions, but I would lean toward something more straightforward like:

import Data.List (subsequences)
f :: (Num a, Ord a) => [a] -> a
f = maximum . map product . filter ((==3) . length) . subsequences

Thinking about functional algorithms as sequences of transformations on collections makes them much more idiomatic than transforming imperative loops into recursive functions.

Note if you are doing this with really long lists where efficiency is a concern, you can sort the list first, then take the lowest two and the highest three, and the algorithm will still work:

takeFirstLast xs = (take 2 sorted) ++ (drop (length sorted - 3) sorted)
  where sorted = sort xs

However, my original way is plenty fast up to lists of size 100 or so, and is a lot easier to understand. I don't believe in sacrificing readability for speed until I'm told it's an actual requirement.