Undefined behavior in replacement for polymorphism with out Pointers

I've recently asked about this on this question. And I've gone with this generic approach:

#define COFFEE_GEN_GENERIC_VALUE(CLASS_TYPE) using GenericValue##CLASS_TYPE = coffee::utils::GenericValue<CLASS_TYPE, COFFEE_GENERIC_VALUE_CONFIG_MAX_DERIVED_SIZE_OF_##CLASS_TYPE>

template<typename Base, size_t MaxDerivedSize>
class GenericValue
{
private:
    char buffer[MaxDerivedSize];

public:
    GenericValue() = default;

    template<typename Derived, typename... ConstructorArgs>
    static GenericValue<Base, MaxDerivedSize> create(ConstructorArgs... args)
    {
        static_assert(std::is_base_of<Base, Derived>::value, "Derived must derive from Base.");
        static_assert(sizeof(Derived) <= MaxDerivedSize, "The size specified with the macro COFFEE_GenericValueMaxSize is to small.");
        GenericValue<Base, MaxDerivedSize> result{};
        new ((Derived*) result.buffer) Derived{args...};
        return result;
    }

    template<typename Derived>            
    static GenericValue<Base, MaxDerivedSize> from(Derived* toCopy)
    {
        static_assert(std::is_base_of<Base, Derived>::value, "Derived must derive from Base.");
        static_assert(sizeof(Derived) <= sizeof(MaxDerivedSize), "The size specified with the macro COFFEE_GenericValueMaxSize is to small.");
        GenericValue<Base, MaxDerivedSize> result{};
        memcopy(result.buffer, toCopy, sizeof(Derived));
        return result;
    }
    
    GenericValue(const GenericValue<Base, MaxDerivedSize>& other)
    {
        memcopy(buffer, other.buffer, MaxDerivedSize);
    }
    
    GenericValue(GenericValue<Base, MaxDerivedSize>&& other)
    {
        memcopy(buffer, other.buffer, MaxDerivedSize);
    }

    GenericValue<Base, MaxDerivedSize>& operator=(const GenericValue<Base, MaxDerivedSize>& other)
    {
        memcopy(buffer, other.buffer, MaxDerivedSize);
        return *this;
    }
    
    GenericValue<Base, MaxDerivedSize>& operator=(GenericValue<Base, MaxDerivedSize>&& other)
    {
        coffee::utils::copy(buffer, other.buffer, MaxDerivedSize);
        return *this;
    }

    Base& operator*()
    {
        return *(Base*)buffer;
    }

    Base* operator->()
    {
        return (Base*)buffer;
    }
}

I had the idea for this form this website and its kind of the same as the awnser from @Matthias Grün on my old question. You use the COFFEE_GEN_GENERIC_VALUE macro to create a typedef for GenericValue<Base, MaxDerivedSize> the size will be passed to it with a second macro. This is so that my library can generate the appropriate GenericValue types that will be used for the specific base class. The basic idea behind it is that you store the whole instance of the derived type with it's vtable in buffer. You can than copy it just like a regular rvalue.

Unfortunately doe copying only some times works and I have no idea why, based on my knowledge this would not be a problem. If I remove the copy operators and constructor and just use the class as a wrapper for new it works great (except the objects aren't realy copied).

Does any body know what the problem could be?

I will post a specific example when I found an edge case for which I don't have to post my whole project.