Correct way to pass GPIO port as template argument

From the avr-libc FAQ, we we read that the correct way to pass PORTX as a parameter is with a pointer (or with c++ a reference), so this works:

void set_bits(volatile uint8_t &port, uint8_t mask) {
  port |= mask;
}

int main() {
  set_bits(PORTB, 0xFF);
}

Another thing that works correctly is variable references/pointers in templates, so this works as well:

template<volatile uint8_t &CHANGEME>
class Changer {
public:
  void change() {
    CHANGEME = 4;
  }
};

int main() {
  uint8_t value = 5;
  Changer<value> changer;
  changer.change();
}

And will get correctly optimized away into just two instructions, by the way. What doesn't work is combining those two ideas:

template<volatile uint8_t * CHANGEME>
class PortChanger {
public:
  void change() {
    CHANGEME = 4;
  }
};

int main() {
  uint8_t value = 5;
  Changer<&PORTB> changer;
  changer.change();
}

to which the compiler says:

error: '(24 + 32)' is not a valid template argument for 'volatile uint8_t*' because it is not the address of a variable

Fine, I understand, those are defined as simple macros, after the preprocessor does it's thing, the code looks something like this, I guess:

int main() {
  uint8_t value = 5;
  Changer<&(*(volatile uint8_t *)((0x18) + 0x20))> var;
  changer.change();
}

I tried a few more things, but it seems that I cannot force the compiler to just accept my intent.

I understand I can pass the port to the constructor and in most cases, everything will be optimized away as well (I tried), but out of curiosity I want to know if it is possible to enforce compile-time port this way.
We are not looking for code strictly conforming to standards. "Good enough" is good enough.
What are your ideas?