Passing an IO register as template parameter

I want to use an IO-Register ( == static memory address ) as a template parameter. The problem is, that registers are commonly defined as macros expanding to something similar to (*(volatile uint8_t*)(11 + 0x20)) which I somehow cannot get to work with my templates properly.

I would like to write code like:

Foo<PORTB> foo;

That way I can easily change the IO register the class uses without any runtime overhead (which is crucial for microcontrollers). I have included a full example below:

#include <stdint.h>
#include <stdio.h>
#include <utility>

#define PORTB  (*(volatile uint8_t*)(11 + 0x20))

template<volatile uint8_t* PortRegister>
class ControllerPtr final
{
public:
    static void SetHigh() { *PortRegister |= 0x2; }
};

template<volatile uint8_t& PortRegister>
class ControllerRef final
{
public:
    static void SetHigh() { PortRegister |= 0x2; }
};


int main()
{
    ControllerPtr<&PORTB> ptr;
    ControllerRef<PORTB> ref;

    ptr.SetHigh();
    ref.SetHigh();

    // Both statements should be equal to:
    // PORTB |= 0x2;
}

Everytime when I try to pass &PORTB to ControllerPtr, g++ fails to compile:

error: (volatile uint8_t*)((long int)(11 + 32)) is not a valid template argument for volatile uint8_t* {aka volatile unsigned char*} because it is not the address of a variable

error: expression *(volatile uint8_t*)((long int)(11 + 32)) has side-effects

The error is a little bit different when trying to pass PORTB to a reference type like used in ControllerRef:

error: *(volatile uint8_t*)((long int)(11 + 32)) is not a valid template argument for type volatile uint8_t& {aka volatile unsigned char&} because it is not an object with linkage

I actually don't understand why this error is an error, as I don't see any problem with passing static addresses to templates.