Destructor throwing exception because of memset

I have a piece of legacy code that I know is the culprit of this exception, but I'm trying to understand why. Magically, if I change the container then the exception goes away. I have learned in the past that memsetting an stl::container can be dangerous. Why does memsetting it cause undefined behavior.

In our code we have this. (I apologize for pseudo code, but the point should still be understandable...)

struct STRUCT_A
{
    B b;
}

class B
{
    std::vector<CustomObject> vect;
}

Later on we do this...

STRUCT_A m_struct_a;
memset(&m_struct_a, 0, sizeof(STRUCT_A));

This works perfectly fine even if we memset an stl::container! However, if we change class B to have a map then an exception occurs on the destructor.

class B
{
    std::map<CustomObject> vect;
}

So I thought it had to do with the vector being contiguous, so I changed it to an unordered_map

class B
{
    std::unordered_map<CustomObject> vect;
}

An exception is still thrown on the destructor. Yes I understand memsetting an object causes undefined behavior, but why does it work for a vector and not the other containers? I thought it was pretty interesting, and thought it was a good question to ask...