Template friendship based on inheritance relationship

I'm not sure if this is achievable. Suppose I have two class templates Foo<T> and FooOwner<T>, and a FooOwner<T> has a pointer member to a Foo<U>, where U is a subclass of T.

I want to make FooOwner<T> a friend of Foo<U>, but only if U is a subclass of T. Is that possible? If not, is there any workaround for something close enough?

template<typename T>
class Foo {
private:
    template<typename U>  // Constrain U?
    friend class FooOwner;
};