Using `std::weak_ptr` after DLL unload

I have a problem with using std::weak_ptr to an object from another DLL after it was unloaded.
Here is the simplified situation: there is an application that loads 2 DLLs. DLLs provide some objects, that inherit common interface. The application keeps a kind of cache of these objects (a std::vector of std::shared_ptrs), so that DLLs may use each other objects indirectly.
When DLL unloads it removes it's objects from cache.

But the problem comes if another DLL keeps a weak_ptr to another DLL's object, because when the destructor of thar weak_ptr is called, it tries to delete the control block, and that causes an AV error, since the owner DLL is already unloaded.
I though I may enforce control block to be allocated in the application, instead of DLL, but I could not figure out how to do it. I've tried creating an Allocator on the application side and passing it to share_ptr constructor, but the control block still keeps some pointer to DLL's classes which, again, cause AV errors.

The only thing that seems to work is to create raw pointers to objects in DLLs and then pass them to the application, which than wraps them in shared_ptr. But that leads to some problems with enable_shared_from_this.

So, my question is - what is the best thing to do?