Proper destructors for unions

I'm trying to use an union (C++) that has some non-primitive variables, but I'm stuck trying to create the destructor for that class. As I have read, it is not possible to guess what variable of the union is being used so there is no implicit destructor, and as I'm using this union on the stack, the compiler errors that the destructor is deleted. The union is as follows:

struct LuaVariant {
    LuaVariant() : type(VARIANT_NONE) { }

    LuaVariantType_t type;
    union {
        std::string text;
        Position pos;
        uint32_t number;
    };
};

The type variable holds what field of the union is being used (chosen from an enum), for the purpose of reading from the union and it could be used to guess what value should be deleted. I tried some different approaches but none of them worked. First of all, just tried the default destructor:

~LuaVariant() = default;

It did not work, as the default is... deleted. So, I tried swapping the value with an empty one, so that the contents would be erased and there would be no problem "leaking" an empty value:

~LuaVariant() {
    switch (type) {
        case VARIANT_POSITION:
        case VARIANT_TARGETPOSITION: {
            Position p;
            std::swap(p, pos);
            break;
        }
        case VARIANT_STRING: {
            std::string s;
            std::swap(s, text);
            break;
        }
        default:
            number = 0;
            break;
    }
};

But as I am not a master of the unions, I don't know if that can cause other problems, such as allocated memory that never gets deallocated, or something like that. Can this swap strategy be used without flaws and issues?