Is it possible to modify a value that is being pattern-matched on?

In the following program, I attempt to create a change function which mutably modifies the constructor of a datatype:

enum Typ {
    Foo { x: u32 },
    Bar { x: u32 },
}
use Typ::*;

fn change(val: &mut Typ) {
    match val {
        &mut Foo { ref mut x } => *val = Bar { x: *x },
        &mut Bar { ref mut x } => *val = Foo { x: *x },
    }
}

fn main() {
    let mut val = Foo { x: 1 };
    change(&mut val);
    println!("Hello, world!");
}

That does not work:

error[E0506]: cannot assign to `*val` because it is borrowed
 --> src/main.rs:9:35
  |
9 |         &mut Foo { ref mut x } => *val = Bar { x: *x },
  |                    ---------      ^^^^^^^^^^^^^^^^^^^^ assignment to borrowed `*val` occurs here
  |                    |
  |                    borrow of `*val` occurs here

error[E0506]: cannot assign to `*val` because it is borrowed
  --> src/main.rs:10:35
   |
10 |         &mut Bar { ref mut x } => *val = Foo { x: *x },
   |                    ---------      ^^^^^^^^^^^^^^^^^^^^ assignment to borrowed `*val` occurs here
   |                    |
   |                    borrow of `*val` occurs here

Is it possible to modify something inside its pattern-matching clause?