Application using wrong template function definition in C++ file

I have a class that stores a specific variable, and I use templates to specify the type. This is important, because OpenGL has different functions to update these values in the shader. Here's the header shader_variable.h.

template<typename T, typename = std::enable_if<std::is_pointer<T>::value == false>>
class ShaderVariable : public Base::ShaderVariable
{
public:
    ShaderVariable(const std::string& a_Name, ShaderProgram& a_Program) :
        Base::ShaderVariable(a_Name, (Liane::Base::ShaderProgram&)a_Program, (void*)&m_Value),
        m_Name(a_Name)
    {
    }

    operator const T&() const { return m_Value; }
    const T& GetValue() const { return m_Value; }


    void Update() override;
private:
    std::string m_Name;
    T m_Value;
};

in the CPP file, I have the following definition

template<> void ShaderVariable<glm::vec3>::Update()
{
    std::cout << "Updating vec3" << std::endl;
    glUniform3fv(m_Reference, 1, glm::value_ptr(m_Value));
}

template<> void ShaderVariable<glm::vec4>::Update()
{
    std::cout << "Updating vec4" << std::endl;
    glUniform4fv(m_Reference, 1, glm::value_ptr(m_Value));
}

template<> void ShaderVariable<glm::mat3>::Update()
{
    std::cout << "Updating mat3" << std::endl;
    glUniformMatrix3fv(m_Reference, 1, GL_FALSE, glm::value_ptr(m_Value));
}

template<> void ShaderVariable<glm::mat4>::Update()
{
    std::cout << "Updating mat4" << std::endl;
    glUniformMatrix4fv(m_Reference, 1, GL_FALSE, glm::value_ptr(m_Value));
}

And in the main function, I make a glm::mat4 ShaderVariable first, and a glm::vec3 second. If I do this, everything compiles fine, and all ShaderVariables call the glm::mat4 Update().

Now, usually I put my template definitions in the header file, but due to the complexity I prefered the cpp file instead. Is there a way to mitigate this issue?

I also added template class ShaderVariable<glm::vec3>; template class ShaderVariable<glm::vec4>; template class ShaderVariable<glm::mat3>; to the bottom, as another question's answer suggested it, but this does nothing for my issue.