Compile time error if brace-closed list is the wrong size for class constructor

I'm trying to write a class based around mathematical vectors:

template <unsigned N> class Vector{
public:
    Vector() = default;
    Vector(std::initializer_list<double> li) { *this = li;}
    Vector& operator=(std::initializer_list<double>);

private:
    std::array<double, N> x = {}
}

template <unsigned N> inline Vector<N>& Vector<N>::operator=(std::initializer_list<double> li){
     if(N != li.size()) throw std::length_error("Attempt to initialise Vector with an initializer_list of different size.");
     std::copy(li.begin(), li.end(), x.begin());
     return *this;
}

I want to be able to write code like this;

Vector<3> a = {1,2,3};
a = {3,5,1};

It would be natural for a user to expect to write code like that, right? However I want compile-time errors to occur if I use the wrong-sized initializer list, much like std::array does.

 std::array<double, 3> a = {2,4,2,4} //compile time error
 Vector<3> a = {3,5,1,5} //run-time error as of right now

My first idea was to use std::array as the constructor/operator parameter so implicit conversions would occur and then the constructor would hijack, from std::array, the compile time errors. Except array is an aggregate class and works through brace-initialisation rather than defining any constructors/operators of its own, so implicit conversions will not occur.

I thought maybe to use a Variadic member template:

template <typename... Args> Vector(Args... li): x({li...}){
    static_assert(sizeof...(li) == N);
}

It has to be typename... rather than double... because nontype parameters must be integral types. But then I run in to a narrowing conversion error

Vector<2> a = {3,2} //error: narrowing conversion of 'li#0' from 'int' to 'double' inside { } [-Wnarrowing]|
 //error: narrowing conversion of 'li#1' from 'int' to 'double' inside { } [-Wnarrowing]|

Presumably for violating [8.5.4]/7

A narrowing conversion is an implicit conversion

— from an integer type or unscoped enumeration type to a floating-point type, except where the source is a constant expression and the actual value after conversion will fit into the target type and will produce the original value when converted back to the original type, or

The parameters from expanding li... aren't constant expressions and hence produce the narrowing conversion error. As far as I'm aware it wouldn't even be possible to make function parameters as constant expressions (nor would it make much sense?). So I'm not sure how to carry on down that route. Obviously Vector<2> a = {2.,3.} works fine but this puts a burden on the user to remember only to supply floating-point literals.