Combine multiple class template specializations

Given the example below, what's the best way to condense the template specializations such that one or two instructions would suffice to define all of the special values? Variadics, perhaps?

enum class PositiveDigit
{ Zero, One, Two, Three, Four, Five, Six, Seven, Eight, Nine };

// Base class template
template <std::underlying_type_t<PositiveDigit>>
struct IsNum
{ static constexpr bool aPrimeDigit = false; };

// Specialized class templates to define which positive digits are prime 
// HOW TO BEST COMBINE/CONDENSE THESE? VARIADIC?? Ideally, something like:
//    template <PositiveDigit::One, PositiveDigit::Two, ……> ……   OR,
//    template <> …… (PositiveDigit::One, PositiveDigit::Two, ……) ……   OR??
template <>
struct IsNum<static_cast<std::underlying_type_t<PositiveDigit>>(PositiveDigit::One)>
{ static constexpr bool aPrimeDigit = true; };
template <>
struct IsNum<static_cast<std::underlying_type_t<PositiveDigit>>(PositiveDigit::Two)>
{ static constexpr bool aPrimeDigit = true; };
template <>
struct IsNum<static_cast<std::underlying_type_t<PositiveDigit>>(PositiveDigit::Three)>
{ static constexpr bool aPrimeDigit = true; };
template <>
struct IsNum<static_cast<std::underlying_type_t<PositiveDigit>>(PositiveDigit::Five)>
{ static constexpr bool aPrimeDigit = true; };
template <>
struct IsNum<static_cast<std::underlying_type_t<PositiveDigit>>(PositiveDigit::Seven)>
{ static constexpr bool aPrimeDigit = true; };

int main() {
    // Note: It's perfectly okay to pass integers beyond the range of the
    //  enum class, they'll simply provide a false result
    IsNum<-5>::aPrimeDigit; // false
    IsNum<13>::aPrimeDigit; // false
    IsNum< 7>::aPrimeDigit; // true!
}

Please assume as given that the enum must remain strongly-typed. The real-world issue has a large enum class, many potential specializations, and has nothing whatsoever to do with digits or primes; this was just a simple example.

These similar questions don't appear to address the issue at hand (unless I'm missing something):

• Template specialization with multiple variadic templates
• Group class template specializations
• SFINAE and partial class template specializations