Where to delete an object created by factory?

If I have a factory, that creates an object and returns a pointer to it, what will be a better way to delete it:

By delete call in the "user" code, or by a new DestructObject function which I should have together with the factory?