Move-assignment and reference member

Copy-assignment for a class with a reference member variable is a no-no because you can't reassign the reference. But what about move-assignment? I tried simply moveing it but, of course, that destroys the source object when I just want to move the reference itself:

class C
{
public:
    C(X& x) : x_(x) {}
    C(C&& other) : x_(std::move(other.x_)) {}
    C& operator=(C&& other)
    {
        x_ = std::move(other.x_);
    }
private:
    X& x_;
};

X y;
C c1(y);

X z;
C c2(z);

c2 = c1; // destroys y as well as z

Should I just not be implementing move-assignment and sticking with move-construction only? That makes swap(C&, C&) hard to implement.