Writing a template function for basic and Eigen data types

How can I write a function that takes both fundamental data types (int, float, double, …) and Eigen library types (Vector2f, Vector4d, Matrix4f, …)? Specifically, I want a cast function that casts the provided parameter to type N.

For example:

float x1 = cast<float>(1);
double x2 = cast<double>(2.0);
Vector2d x3 = cast<double>(Vector2f(3.0f, 3.0f));
Vector2f x4 = cast<float>(Vector2i(4, 4));

The easy part:

template<typename T, typename N>
N cast(const T& source) const
{
    return static_cast<N>(source);
}

Casting an Eigen type:

template<typename T, typename N>
Eigen::CwiseUnaryOp<Eigen::internal::scalar_cast_op<typename Eigen::internal::traits<T>::Scalar, N>, const T> cast(const Eigen::MatrixBase<T>& source) const
{
    return source.cast<N>();
}

In Eigen, a cast from Vector2f v to Vector2d is done with v.cast<double>(), so the template parameter is the data type of the scalar, not the new type itself.

The trouble that I have (at least I think that's the main issue) is that I don't know how I can put these two templates together. The Eigen one should probably be a specialization of the first one, but is this even possible? The templates themselves compile, but e.g. cast<Vector2f, double>(Vector2f::Zero()) will not, because 'static_cast' : cannot convert from 'const Eigen::Vector2f' to 'double'.

What to do? C++11 solutions are very welcome, but please type slowly, as I am no template wizard.