Program Debug Database files in a Release output folder

I've switched over my solution's Build configuration to Release mode and now have an output in the "Release" folder instead of in the "Debug" one.
But still the Release output contains Program Debug Database files. Why is that? When should I keep them, when and how should I get rid of them?