Why is iterating over a std::set so much slower than over a std::vector?

While optimizing performance critical code, I noticed iterating over a std::set was a bit slow.

I then wrote a benchmarker and tested the speeds of iteration over a vector by iterator (auto it : vector), iterating over a set by iterator, and iterating over a vector by index (int i = 0; i < vector.size(); ++i).

The containers are constructed identically, with 1024 random ints. (Of course, each int is unique since we're working with sets). Then, for each run, we loop through the container and sum their ints into a long int. Each run has 1000 iterations doing the sum, and the test was averaged over 1000 runs.

Here are my results:

Testing vector by iterator
✓           
Maximum duration: 0.012418
Minimum duration: 0.007971
Average duration: 0.008354

Testing vector by index
✓           
Maximum duration: 0.002881
Minimum duration: 0.002094
Average duration: 0.002179

Testing set by iterator
✓           
Maximum duration: 0.021862
Minimum duration: 0.014278
Average duration: 0.014971

As we can see, iterating over a set by iterator is 1.79x slower than by vector, and a whopping 6.87x slower than vector by index.

What is going on here? Isn't a set just a structured vector that checks whether each item is unique upon insertion? Why should it be so much slower?