The measure function does not return a Bool. In both of the examples you gave, the measure function returns an Int. Also, the final return value of minBy is not necessarily an Int. It is a value with the same type as the elements of the list. So if you give the minBy function a list of strings, the returned value will be a string. With these two considerations in mind, the type signature of minBy should be:
minBy :: (a -> Int) -> [a] -> a
This is not the most general type, however. The value returned by the measure function doesn't have to be an Int, it can also be a Double or a String or anything comparable. The way to express that a value is comparable is with the Ord constraint.
minBy :: (Ord b) => (a -> b) -> [a] -> a
In the first branch of the case expression, you define the minimum value of an empty list to be 0. This definition will fail to type-check. The minimum value of a list must have the same type as the elements of the list, and the type annotation for minBy states that you can give a list of any type, not just a list of integers.
Defining the minimum value of an empty list to be 0 also conflicts with the definition of "minimum value." The empty list doesn't have a minimum value; it has no values at all! A better base case would be: "if the list has just one element, the minimum value is that element."
case list of
[x] -> x
The second branch of your case expression is almost correct; you just need to replace > with <. The first element should be kept if its measure is smaller than the second element's. You also need to surround x:rest in parentheses, because the : operator has lower precedence than function calls. minBy measure x:rest will be parsed as (minBy measure x) : rest, which is not what you want.
minBy :: (Ord b) => (a -> b) -> [a] -> a
minBy measure list =
case list of
[x] -> x
x:y:rest ->
if measure x < measure y
then minBy measure (x:rest)
else minBy measure (y:rest)
I also replaced the variable you called xs with y, because the name xs in Haskell is normally used for lists.