std::set of std::weak_ptr insert and remove

In my main application, I have an std::set<std::shared_ptr<Object>> but I also want to create a secondary set of std::set<std::weak_ptr<Object>> because I don't want to increment the reference count but I also want to avoid calling bad memory locations by utilising lock. Except std::set doesn't take weak_ptr and throws compiler errors in the insert and remove method.

#include <memory>
#include <set>
#include <vector>
class Test{
private:
    int x ;
public:
    Test(int x){
        this->x = x;
    }
    int getx(){
        return x;
    }
};
int main(){
    std::shared_ptr<Test> t = std::make_shared<Test>(10);
    std::weak_ptr<Test> ref = std::weak_ptr<Test>(t);
    std::set<std::weak_ptr<Test>> weakrefs;
    weakrefs.insert(ref);//compiler error
    weakrefs.erase(ref);//compiler error
}

But when I use std::vector>, I'm allowed to push back and pop back. But I would like similar functionality to std::set..