Avoid exponential grow of const references and rvalue references in constructor

I am coding some templated classes for a machine learning library, and I'm facing this issue a lot of times. I'm using mostly the policy pattern, where classes receive as template argument policies for different functionalities, for example:

template <class Loss, class Optimizer> class OptimizableLinearClassifier { ... }

The problem is with the constructors. As the amount of policies (template parameters) grows, the combinations of const references and rvalue references grow exponentially. In the previous example:

LinearClassifier(const Loss& loss, const Optimizer& optimizer) : _loss(loss), _optimizer(optimizer) {}

LinearClassifier(Loss&& loss, const Optimizer& optimizer) : _loss(loss), _optimizer(optimizer) {}

LinearClassifier(const Loss& loss, Optimizer&& optimizer) : _loss(loss), _optimizer(optimizer) {}

LinearClassifier(Loss&& loss, Optimizer&& optimizer) : _loss(loss), _optimizer(optimizer) {}

Is there some way to avoid this?