Find a pointer in a std::set based on a const pointer

The following code does not work because of the marked line, since s.find expects an int*, not const int*. Since nothing is being changed about the variable b, why is it not possible to call the method like that? Is there a nicer way than const_cast?

#include <set>

int main()
{
    int x = 5;
    int* a = &x;
    const int* b = a;

    std::set<int*> s;
    s.insert(a);
    s.find(b); // does not work
    s.find(const_cast<int*>(b)); // does work, but const_cast
}