Cannot return vector containing derived pointers

Okay, so to illustrate the issue I am having I will show some (pseudo) code.

Lets say I have the following models:

class Animal : public GameObject;

class Player : public GameObject;

class GameObject : public ObjectInterface;

class ObjectInterface
{
public:
    virtual ~ObjectInterface() = default;
    virtual vec3 GetPosition() = 0;
}

Now I also hold some "object context", which holds collections of certain game objects.

class ContextObject
{
     // they implement ObjectInterface
     vector<shared_ptr<Animal>> animals;
     vector<shared_ptr<Player>> players; 
}

Now I have a TargetSelector class, which works directly with the ObjectInterface only.

class TargetSelector
{
    // this is somehow not possible, although `animals` are a subclass of `ObjectInterface`
    vector<shared_ptr<Model::ObjectInterface>>& GetAvailableTargets()
    {
        return context->animals; // context is some `ObjectContext`
    }
}

I would expect the above code to work, since an Animal is of the type ObjectInterface. But instead I get an error saying it cannot convert from an vector<shared_ptr<Animal>> to an vector<shared_ptr<ObjectInterface>>. Is this even suppose to work?

Could someone explain me why I cannot do this kind of polymorphism and if possible a nice solution so I can make this work.

Thanks, any help is appreciated!