C++ std::unordered_set find/count/contains const key reference

I have a std::unordered_set<int *> (in my real code I use pointers to a class, but int * works for this example too) and want to check whether a given pointer is stored in that set. Hence the function contains will only be available for C++20, I'm using the function size_type count( const Key& key ) const.

If I'm now searching for a const pointer instead of a pointer, the compilation fails with the message error: invalid conversion from ‘const int*’ to ‘std::unordered_set<int*>::key_type {aka int*}’ [-fpermissive]

Example code:

#include <unordered_set>

int main() {
    std::unordered_set<int *> set {};

    int foo = 42;
    set.insert(&foo);

    int *pointer = &foo;
    set.count(pointer); // works fine

    const int *const_pointer = pointer;
    set.count(const_pointer); // doesn't work
    set.count(const_cast<int *>(const_pointer)); // works fine

    return 0;
}

I'm using g++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 with C++11.

Is there a way to avoid that ugly const_cast<>? That error seems quite unimportant to me...