Self releasing (reference counting) singelton

Consider the following scenario: You have a singleton class that represent some sort of data provider. This singleton class allocates a lot of memory,and you want it to release it's allocated memory when there is no one using him. Flow:

1. Class A call getInstance and uses singleton (this is the first time getInstance called and singleton class allocates huge memory chunk)
2. Class B call getInstance and uses singleton
3. Class A and class B "dies" (no one using singleton now)
4. Program still running but singleton's memory is not released.

How do you suggest implementing singleton that at stage 3 (class A and B "dies") will free the memory (I know that java uses garbage collection but still lets say I want the following memory = null).

PS I don't want to force each class that uses the singleton call release on singleton whenever it stops using it. I want the singleton to handle "releasing" memory by himself.