Initializing a tuple's elements with non-const references to another tuple's elements of different type

I'm trying to do a conversion initialization of an std::tuple (called w) with elements from another std::tuple (called t) with same size but different element types.

In my application, the elements of w must be constructed using a non-const reference to the respective element of t.

Unfortunately, there only exists a 4) converting copy-ctor and a 5) converting move-ctor for std::tuple (reference). This means, the new element from w can only be initialized using either a const reference or rvalue reference to the element of t, and both are not applicable.

Example

First we have a element modifier class, that only binds to references of its element type.

template<typename T>
struct SingleElementModifier {
    T& ref;
    SingleElementModifier(T& ref) : ref(ref) { }
};

Next a type trait to get the type of w given the type of t. Essentially it wraps each element from t into the SingleElementModifier.

template<typename T> struct ModifierTuple;
template<typename... Ts>
struct ModifierTuple<std::tuple<Ts...>> {
    using Type = std::tuple<SingleElementModifier<Ts>...>;
};

Now we have a TupleModifier that takes a tuple t and stores a tuple w with modifiers to each element of the first tuple.

template<typename Tuple>
struct TupleModifier {
    using TupleType = typename ModifierTuple<Tuple>::Type;
    TupleType w;
    TupleModifier(Tuple& t) : w{ t } { }
};

Now we only need to choose some tuple t and create the modifer.

int main() {
    /* just some example types, could be any type inside. */
    std::tuple<double, std::pair<int, int>> t;
    TupleModifier mod(t);
}

Live example

Here the compiler will stop and complain to me that no constructor for w{ t } could be found, for the reason explained in the beginning.

Bad workaround

Theoretically, I could have SingleElementModifier take a const T& and use const_cast to get rid of the const, but I was hoping for a better solution that doesn't require me to throw away const-correctness.