Why is it hard to unit test a system that depends on singletons?

I've read cases for and against using the singleton pattern. One common case against describes the difficulties in unit testing with singletons, but I'm unclear as to why this is? If the unit test is part of the build couldn't you just reference the singleton and use it when its needed? (I'm thinking from a java perspective but I guess it shouldnt matter)