Swapping indices of two elements in a std::vector

Assuming a vector of objects, is it possible to swap the indices of two elements? Swapping two elements at positions i and j is straightforward:

tmp=vec[i];
vec[i]=vec[j];
vec[j]=tmp;

The above, however, makes use of the assignment operator = and depends on the objects stored in vec and the way the assignment operator is defined. In addition, it involves the creation of a new object tmp so I assume that pointers directed to the vector elements could be invalidated. My question is, is it possible to keep the objects stored at i and j and only swap their indices, so that after the operation vec[i] points to the object originally found at vec[j] and the other way around?