Memory management with std::vector

At the moment, I'm using a std::vector of std::unique_ptr.

I push state objects created with std::make_unique using std::move when I want to change state and then I erase the stack.

I'm testing things out using 2 states: one has some textures in it and the other one is empty.

What happens when I repeatedly switch between states is that memory usage increases by 5mb per switch.

I also tried making the textures raw pointers. When I manually delete them before switching states there is no increase in memory. But if I try to make them shared pointers or unique pointers the memory will still increase.

I'm curious how should I handle these states and pointers.