Allow "compilation" of invalid template instantiation if you'll never run it?

I have a function to byteswap and type convert data (think integer to float), I use void pointers so I can take advantage of type erasure:

// if Tfrom converts to Tto, then perform a byteswap and convert to Tto
template <typename Tfrom, typename Tto>
static void convert_swap(void* dptr_, const void* sptr_, ssize_t len) {
          Tto*   dptr =         (Tto*)dptr_;
    const Tfrom* sptr = (const Tfrom*)sptr_;

    for (ssize_t ii=0; ii < len; ii++) {
        dptr[ii] = Tto(byteswap(sptr[ii]));
    }
}

Where byteswap is expected to be defined for Tfrom and Tto(Tfrom) should be well defined.

I use this to read from files that can be one of several types (for sanity, I'll limit them to real integer/float, complex integer/float and other). These are represented by a two-byte format string, "SI", "SF", "CI", "CF", "NH" (non-homogenous).

So, I open my file, and I can create a "reader" on it that handles reading the data from the file and converting it to a type you ask for:

template <typename Tout>
reader<Tout> file::make_reader() {
         if (format() == "SI") return create_reader<Tout,int32_t>(this);
    else if (format() == "SF") return create_reader<Tout,float>(this);
    else if (format() == "CI") return create_reader<Tout,cint32_t>(this);
    else if (format() == "CF") return create_reader<Tout,cfloat>(this);
    else if (format() == "NH") return create_reader<Tout,Tout>(this);
    else 
      assert(false && "saw invalid format string");
}

Where in the generic 'NH' case, I just want to read the type from the file directly (assume the reader handles this). This is nice because you can write your code with a single type in mind and it "just works".

So, in general, with numeric types, this works great. My problem comes with the 'NH' case, where I want to read a struct, let's call it struct some_packet_type {};

The problem is, most of the combinations of numeric types and some_packet_type are not compatible. I can't eg: convert a cfloat to a some_packet_type.

So, the function will fail to compile because those translations don't exist. But, I'll never call them due to my runtime type check.

My best idea right now is to allow convert_swap to compile even with incompatible types via overloading but throw an assertion at runtime if the function is ever actually used.

Barring that, is there a canonical C++ way to say something like 'please let this compile, I promise to never use it and you can segfault my program or the like if I ever do'?