How to use SFINAE to enable implicitness of explicitness of the conversion operator?

Consider the following code:

// Preamble
#include <iostream>
#include <type_traits>

// Wrapper
template <class From>
struct wrapper
{
    // Implicit conversion
    template <class To, class = typename std::enable_if<
        std::is_convertible<From, To>::value
    >::type>
    constexpr operator To() const noexcept;

    // Explicit conversion
    template <class To, class = typename std::enable_if<
        !std::is_convertible<From, To>::value
        && std::is_constructible<To, From>::value
    >::type>
    explicit constexpr operator To() const noexcept;
};

// Main
int main(int argc, char* argv[])
{
    wrapper<int> x;
    double y = x;
    return 0;
}

Ideally this code would make the conversion operator implicit when From is implicitly convertible to To, and make the conversion operator explicit when To is explicitly constructible from From.

However, the code currently does not compile because from the compiler standpoint, both conversion operators have the same signature.

Question: Would there be any way to trick the compiler to produce the expected behavior?