C++ Conditional Templates Compilation based on Type

I was really hoping I could achieve this using C++ (11) Templates, but I'm facing some problem here. So we have a custom pointer object which internally could be of any of these types: a list of similar objects/an int/a char/bool/long/double/char* or any standard type, and it's decided by a flag which this object stores. There are global methods to get values of specific types from this list.

Now, my purpose is simple. I know that for my case, my object is a list of such objects, so I wanted to write a function like this as it's a common scenario:

template <typename T>
std::vector<T> List_To_Vector(Object& list)
{
    std::vector<T> vec;
    int listSize = Func_Size(list);    
    for (int i = 0; i < listSize; ++i)
    {
        //let's say this function gives list items one by one
        Object arg = Func_Next(list);
        if (std::is_same<T, int>::value || std::is_same<T, unsigned int>::value)
            vec.push_back((T)Func_IntValue(arg));
        else if (std::is_same<T, float>::value || std::is_same<T, double>::value)
            vec.push_back((T)Func_DoubleValue(arg));
        else if (std::is_same<T, std::string>::value || std::is_same<T, char*>::value)
            vec.push_back((T)Func_StringValue(arg)); //Func_StringValue returns char*, so conversion to std::string should work
        else if (std::is_same<T, bool>::value)
            vec.push_back(Func_BoolValue(arg));
        else if (std::is_same<T, char>::value)
            vec.push_back(Func_CharValue(arg));

        vec.push_back(val);
    }

    return vec;
}

int main()
{
    Object listContainingStrings = GetListOfNames();
    Object listContainingNumbers = GetListOfNumbers();
    std::vector<string> vec1 = List_To_STD_Vector<string>(listContainingStrings);
    std::vector<int> vec2 = List_To_STD_Vector<int>(listContainingNumbers);
    return 0;
}

The problem is, C++ complains here as it tries to compile the code taking T = std::string, and int to string or float to string conversions would fail. What I really wanted here was a way to compile the int part of the code when the type is detected as int, and not of any other type. I can use Template function specialization or overloading, but then I think it really defeats the purpose of templates here and I could just write 8 different functions for 8 different types (e.g. List_To_String_Vector, List_To_Int_Vector and so on).

I also tried another hack, using reinterpret_cast on the address of the each return types, and then dereferencing it to add to the vector. That kind of worked, but has Compiler warnings and I think that's undefined behavior.

Is there a way to make this work properly?

Thanks!