Can I disable a specialization of std::char_traits based on a boolean expression?

Using C++11 (or C++14), I'd like to specialize std::char_traits for unsigned char, but only if !std::is_same<char, unsigned char>::value. (Since std::char_traits<char> already exists.)

So I currently have such a specialization, which begins like this:

namespace std {
template<>
struct char_traits<unsigned char> {
    using char_type = unsigned char;
    etc...

The question is if there is a way I can make this specialization "not exist" via SFINAE if std::is_same<char, unsigned char>::value. All the examples I can find for enabling/disabling specializations depend on the non-specialized template having a dummy template parameter for enabling. I obviously can't change std::char_traits in this way. Is there a path I am missing?