One template specialization for several enum values

Normally if I want to have a templated (data) class by enum I would write something like this

enum class Modes : int
{
    m1 = 1,
    m2 = 2,
    m3 = 3
};

template <Modes M>
class DataHolder
{
};

template<>
class DataHolder<Modes::m1>
{
    public: int a = 4;
};

Then if I want the same specialization for the Modes::m1 as for the Modes::m2 I would write the same specialization again. Is there a way to write one specialization for several enum values? I have tried it with SFINAE, but I am not succesfull.

template <Modes M, typename = void>
class DataHolder
{
};

template<Modes M, typename = typename std::enable_if<M == Modes::m1 || M == Modes::m2>::type>
class DataHolder
{
    public: int a = 4;
};

This doesn't not compile. Especially, after I would like to carry on with different specialization for Modes::m3. I've tried many similiar solution found here on SO, but nothing seems to be solving the issue.