Remove shared_ptr from set

I am holding several sets of shared pointers that are used throughout my program. Now I could not erase a shared_ptr from the set, since it was copied (it isnt passed by reference) and so the instance is not actually the same as the one in the set. The underlying raw pointer is of course the same. How could I approach this, so that I can delete it properly from the set?

The reason the values are copied are probably because of some dynamic_pointer_cast. I won't be able to pass all shared_ptr's by reference.