Why would the behavior of std::memcpy be undefined for objects that are not TriviallyCopyable?

From http://ift.tt/1pDNXbt:

If the objects are not TriviallyCopyable (e.g. scalars, arrays, C-compatible structs), the behavior is undefined.

At my work, we have used std::memcpy for a long time to bitwise swap objects that are not TriviallyCopyable using:

void swapMemory(Entity* ePtr1, Entity* ePtr2)
{
   static const int size = sizeof(Entity); 
   char swapBuffer[size];

   memcpy(swapBuffer, ePtr1, size);
   memcpy(ePtr1, ePtr2, size);
   memcpy(ePtr2, swapBuffer, size);
}

and never had any issues.

I understand that it is trivial to abuse std::memcpy with non-TriviallyCopyable objects and cause undefined behavior downstream. However, my question:

Why would the behavior of std::memcpy itself be undefined when used with non-TriviallyCopyable objects?