Shared pointer with observer pointers

I have a class Foo, which has a vector of some large classes. The idea is, that an octal tree will be built recursively out of the elements of the vector, and each OctreeNode will have a pointer to few elements of the vector found in Foo. (In the example, just for simplicity, a node will point to only one element of the vector)

class Foo
{
    vector<LargeClass>  mLargeClasses;

    void removeItem(const int index);    //remove an element from the vector at the index
}

class OctreeNode
{
    LargeClass* mLargeClass;
}

One can say, "why bother keeping the vector after the tree is built, and store the objects in the tree itself". True, let's just say, I need to keep vector parallel to the built tree as well.

While the above concept works, I have issues when elements got removed from the underlying vector. In such case, some Octree nodes end up with dangling pointers.

My solution #1: If removeItem function is called, then before it removes the vector element, it first recursively traverse the octal tree, and make all mLargeClass pointer a nullptr which happen to point to that particular vector element. It's ok to have nullptr in the nodes, as I check each time against nullptr, when I access them anyway.

My solution #2: Have the vector store shared_ptrs, and have the OctreeNode store a weak_ptr. I am not fan of this, as each time I access a weak_ptr in the tree, it gets converted to a shared_ptr in the background with all the atomic counter increases. I am not expert on performance testing, but I have a feeling, that it is slower than a simple pointer access with if condition.

Does anybody know any better solutions?

I think the most elegant would be: To have smart pointer which behaves like a shared_pointer, counts, how many other pointer refers to it, keep a record of them, and in case it gets destroyed, it automatically nulls out all other "observer" pointers which refer to it?