How to simplify these function template specializations?

I am working on some legacy code that has the following template defined in a header file:

template<typename T>
std::string convertToString(const T& t);

In the same header file, there are specializations for some user-defined types. In the .cpp file, there are specializations for the types int, unsigned short and unsigned long as follows:

template<>
std::string convertToString<unsigned short>(const unsigned short& s) {
    ... some implementation ...
}

template<>
std::string convertToString<int>(const int & s) {
    ... some implementation ...
}

template<>
std::string convertToString<unsigned long>(const unsigned long& value) {
    ... some implementation ...
}

The implementations don't make use of C++11's std::to_string function. I would like to simplify this code so that for all types that std::to_string supports, namely

int
long
long long
unsigned
unsigned long
unsigned long long
float
double
long double

the template specialization uses the simple implementation (e.g. for int):

template<>
std::string convertToString(const int& n) { return std::to_string(n); }

I could of course write down all template specializations for all the above mentioned types, but that doesn't seem like a nice way to do it. My current best solution to have these template specializations for all types that std::to_string takes is to add the following to my header file:

template<typename T>
std::string convertToString(const T& t);

template <typename T,
          typename = typename std::enable_if<std::is_arithmetic<T>::value>::type>
std::string convertToString(const T& t)
{
    return std::to_string(t);
}

I think this is close to a solution, but not yet as I get the following ambiguity error:

error: call of overloaded 'convertToString(long unsigned int&)' is ambiguous

referring me to the above two template functions as possible candidates. I don't quite understand why the first one is also a possible candidate if the second one exists...

How can I add template specializations for all the types that std::to_string supports, and also still allow for user-defined types T in the convertToString function template? Solution should be restricted to C++11 for now...