SFINAE for true_type and false_type in std::conditional

What's the best way to make this compile?

// Precondition: Dims is either a pointer or std::map.

using T = std::conditional_t<std::is_pointer_v<Dims>,
    std::remove_pointer_t<Dims>,
    typename Dims::mapped_type>;

When Dims is a pointer, I am getting:

error: template argument 3 is invalid

How do I make it work in SFINAE manner, when condition is true?