How can I reliably get an object's address when operator& is overloaded?

Consider the following program:

struct ghost
{
    // ghosts like to pretend that they don't exist
    ghost* operator&() const volatile { return 0; }
};

int main()
{
    ghost clyde;
    ghost* clydes_address = &clyde; // darn; that's not clyde's address :'( 
}

How do I get clyde's address?

I'm looking for a solution that will work equally well for all types of objects. A C++03 solution would be nice, but I'm interested in C++11 solutions too. If possible, let's avoid any implementation-specific behavior.

I am aware of C++11's std::addressof function template, but am not interested in using it here: I'd like to understand how a Standard Library implementor might implement this function template.