Why can't the literal operator be templated normally?

I'm currently working on a compile time expression tree and wanted to use a literal operator to create an expression from a literal. Since all literals are allowed as an expression I wanted to template the literal operator like:

template <typename T>
constexpr auto operator""_ve(T value)
{
    return value_expression(value);
}

This didn't compile so I did some research and found out that the only allowed template is

template <char...> XYZ operator "" _abc();

Why is that? Do I really have to specify each overload manually?