As in almost all other languages, parentheses in Haskell don't really do anything. They help the parser know which things you intended to group together, and that's it; they don't change the meaning of the thing that's inside them, and if the parser could magically guess which way you intended to group things at every choice point we wouldn't need them at all. So all of these patterns behave exactly the same:
x
(x)
((x))
(((((((((x)))))))))
All of these patterns are also exactly the same:
x:xs
(x:xs)
((x:xs))
(((((((((x:xs)))))))))
Sometimes we must use one of the later ones -- like (x:xs) -- instead of the first one -- x:xs -- to inform the parser that we intended these things to be grouped together. But the parentheses don't otherwise change anything about what the pattern means.
Square brackets, on the other hand, are used for constructing lists. They have an actual run-time meaning: they allocate a new data structure in memory and initialize it. For example, 3 is a number, but [3] is a list with a single element, [3,4] is a list with two elements, [[3]] is a list of lists, and so on. The same behavior applies to patterns:
x -- match anything (even a list!) and bind x to its value
[x] -- match a list with a single element, and bind x to that element's value
[x,y] -- match a list with two elements, and bind x and y to those elements' values
[[x]] -- match a nested list; the outer and inner lists both have one element; bind x to the first element's first element
Now we can address your specific example. The pattern x:xs is a pattern that matches any list with at least one element, binding x to that first element and xs to the rest of the list. The pattern (x:xs) does exactly the same thing. The pattern [x:xs], on the other hand, matches a list with exactly one element, because there are square brackets; the only element of that list is then matched against the pattern x:xs, with the behavior described above.
One consequence of this is that x:xs can match lists with any type of element -- say, Int -- but [x:xs] can only match lists whose elements are lists.