Understanding "the trait X cannot be made into an object" for `&mut Box<Self>` parameter

The problem is when you pass it in as a reference, because the inner type may not be well-sized (e.g. a trait object, like if you passed in a Box<Fluffy>) the compiler doesn't have enough information to figure out how to call methods on it. If you restrict it to sized objects (like your TeddyBear) it should compile

trait Scruffy {
    fn scruff_up(self: &mut Box<Self>) -> Box<dyn Scruffy> where Self: Sized;
}

A receiver is the self (&self, &mut self, self: &mut Box<Self> and so on).

Note that the list you cited from the reference lists both Box<Self> and &mut Self, but does not list &mut Box<Self> nor it says that combinations of these types are allowed.

This is, indeed, forbidden. As for the why, it is a little more complex.

In order for a type to be a valid receiver, it needs to hold the following condition:

Given any type Self that implements Trait and the receiver type Receiver, the receiver type should implement DispatchFromDyn<dyn Trait> for itself with all Self occurrences of Self replaced with dyn Trait.

For instance:

&self (has the type &Self) has to implement DispatchFromDyn<&dyn Trait>, which it does.

Box<Self> has to implement DispatchFromDyn<Box<dyn Trait>>, which it does.

But in order for &mut Box<Self> to be an object-safe receiver, it would need to impl DispatchFromDyn<&mut Box<dyn Trait>>. What you want is kind of blanket implementation DispatchFromDyn<&mut T> for &mut U where U: DispatchFromDyn<T>.

This impl will never exist. Because it is unsound (even ignoring coherence problems).

As explained in the code in rustc that calculates this:

The only case where the receiver is not dispatchable, but is still a valid receiver type (just not object-safe), is when there is more than one level of pointer indirection. E.g., self: &&Self, self: &Rc<Self>, self: Box<Box<Self>>. In these cases, there is no way, or at least no inexpensive way, to coerce the receiver from the version where Self = dyn Trait to the version where Self = T, where T is the unknown erased type contained by the trait object, because the object that needs to be coerced is behind a pointer.

The problem is inherent to how Rust handles dyn Trait.

dyn Trait is a fat pointer: it is actually two words sized. One is a pointer to the data, and the other is a pointer to the vtable.

When you call a method on dyn Trait, the compiler looks up in the vtable, find the method for the concrete type (which is unknown at compilation time, but known at runtime), and calls it.

This all may be very abstract without an example:

trait Trait {
    fn foo(&self);
}

impl Trait for () {
    fn foo(&self) {}
}

fn call_foo(v: &dyn Trait) {
    v.foo();
}

fn create_dyn_trait(v: &impl Trait) {
    let v: &dyn Trait = v;
    call_foo(v);
}

The compiler generates code like:

trait Trait {
    fn foo(&self);
}

impl Trait for () {
    fn foo(&self) {}
}

struct TraitVTable {
    foo: fn(*const ()),
}
static TRAIT_FOR_UNIT_VTABLE: TraitVTable = TraitVTable {
    foo: unsafe { std::mem::transmute(<() as Trait>::foo) },
};

type DynTraitRef = (*const (), &'static TraitVTable);
impl Trait for dyn Trait {
    fn foo(self: DynTraitRef) {
        (self.1.foo)(self.0)
    }
}

fn call_foo(v: DynTraitRef) {
    v.foo();
}

fn create_dyn_trait(v: &impl Trait) {
    let v: DynTraitRef = (v as *const (), &TRAIT_FOR_UNIT_VTABLE);
    call_foo(v);
}

Now suppose that the pointer to the value is behind an indirection. I'll use Box<&self> because it's simple but demonstrates the concept best, but the concept applies to &mut Box<Self> too: they have the same layout. How will we write foo() for impl Trait for dyn Trait?

trait Trait {
    fn foo(self: Box<&Self>);
}

impl Trait for () {
    fn foo(self: Box<&Self>) {}
}

struct TraitVTable {
    foo: fn(Box<*const ()>),
}
static TRAIT_FOR_UNIT_VTABLE: TraitVTable = TraitVTable {
    foo: unsafe { std::mem::transmute(<() as Trait>::foo) },
};

type DynTraitRef = (*const (), &'static TraitVTable);
impl Trait for dyn Trait {
    fn foo(self: Box<DynTraitRef>) {
        let concrete_foo: fn(Box<*const ()>) = self.1.foo;
        let data: *const () = self.0;
        concrete_foo(data) // We need to wrap `data` in `Box`! Error.
    }
}

You may think "then the compiler should just insert a call to Box::new()!" But besides Box not being the only one here (what with Rc, for example?) and we will need some trait to abstract over this behavior, Rust never performs any hard work implicitly. This is a design choice, and an important one (as opposed to e.g. C++, where an innocent-looking statement like auto v1 = v; can allocate and copy 10GB by a copy constructor). Converting a type to dyn Trait and back is done implicitly: the first one by a coercion, the second one when you call a method of the trait. Thus, the only thing that Rust does for that is attaching a VTable pointer in the first case, or discarding it in the second case. Even allowing only references (&&&Self, no need to call a method, just take the address of a temporary) exceeds that. And it can have severe implications in unexpected places, e.g. register allocation.

So, what to do? You can take &mut self or self: Box<Self>. Which one to choose depends on whether you need ownership (use Box) or not (use a reference). And anyway, &mut Box<Self> is not so useful (its only advantage over &mut T is that you can replace the box and not just its contents, but when you do that that's usually a mistake).