Optimize away unused parameterized fields

I'm trying to define a parent class for a hierarchy of "codecs" that operate on a memory "membuf" - some of these codecs are purely functional, but some need to have (non-local) side-effects, like setting a bit in a byte somewhere ("flags" below). So, I'd like to have 2 parent classes, essentially, one that has a member flag_type*, and one that doesn't, and I'd like to save the 8 bytes of the flag_type* - I tried to define a second base class, with no template parameters and no member flag_type* but that didn't work. Any idea?

template <typename flag_type =void>
class Codec
{
  public:
    Codec(flag_type* flags =nullptr)
    : _mem(graph.mem()), _flags(flags)
  {}

  protected:
    membuf& _mem;
    flag_type* _flags;
};