Rather than merely searching for a more elegant implementation, it would might help you more to learn an elegant process of searching for an implementation. This should make it simpler to find elegant solutions.
For any function h on lists we have that,
h = foldr f e
if and only if
h [] = e
h (x:xs) = f x (h xs)
In this case your h is filter p for some boolean function p that selects which elements to keep. Implementing filter p as a "simple" recursive function is not too hard.
filter p [] = []
filter p (x:xs) = if p x then x : (filter p xs) else (filter p xs)
The 1st line implies e = []. The 2nd line needs to be written in the form f x (filter p xs) to match the equation of h above, in order for us to deduce which f to plug in the foldr. To do that we just abstract over those two expressions.
filter p [] = []
filter p (x:xs) = (\x ys -> if p x then x : ys else ys) x (filter p xs)
So we have found that,
e = []
f x ys = if p x then x: ys else ys
It therefore follows,
filter p = foldr (\y ys -> if p y then y : ys else ys) []
To learn more about this method of working with foldr I recommend reading
"A tutorial on the universality and expressiveness of fold" by Graham Hutton.
Some added notes:
In case this seems overly complicated, note that while the principles above can be used in this "semi rigorous" fashion via algebraic manipulation, they can and should also be used to guide your intuition and aid you in informal development.
The equation for h (x:xs) = f x (h xs) sheds some clarity on how to find f. In the case where h is the filtering function you want an f which combines the element x with a tail that has already been filtered. If you really understand this it should be easy to arrive at,
f x ys = if p x then x : ys else ys