Why can't I mutably borrow a primitive from an enum?

I would like to be able to obtain references (both immutable and mutable) to the usize wrapped in Bar in the Foo enum:

use Foo::*;

#[derive(Debug, PartialEq, Clone)]
pub enum Foo {
    Bar(usize)
}

impl Foo {
    /* this works */
    fn get_bar_ref(&self) -> &usize {
        match *self {
            Bar(ref n) => &n
        }
    }

    /* this doesn't */
    fn get_bar_ref_mut(&mut self) -> &mut usize {
        match *self {
            Bar(ref mut n) => &mut n
        }
    }
}

But I can't obtain the mutable reference because:

n does not live long enough

I was able to provide both variants of similar functions accessing other contents of Foo that are Boxed - why does the mutable borrow (and why only it) fail with an unboxed primitive?