Value semantics with wrapper around polymorphic class: how to avoid copying to const-ref

I've got a common abstract superclass Base, with a variety of Derived classes. For convenience, I'd like to be able to operate on the base class as a value, rather than via pointers. I can achieve this by creating a Wrapper using the technique described in this answer. The Wrapper stores a unique_ptr<Base> and forwards method calls to that pointer, and has appropriate constructors allowing me to do

Derived derived;
Wrapper wrap = derived;
wrap.callMethodOnDerived();

so the user can pass, return, and store Wrapper without thinking about the underlying pointers. Base remains entirely hidden.

However, this ends up creating copies when they don't need to be. I.e

void fun(const Wrapper&);

fun(derived);                  // makes copy
const Wrapper& view = derived; // makes copy

will call the Wrapper(const Derived&) converting constructor, then passes a reference to that. Instead, I'd like it to create a const Wrapper& which points to the original object, with no copy being made. How can I do this?

#include <memory>

using BigVal = int; // Could be some expensive-to-copy type
using namespace std;

struct Base{
    Base(BigVal x) : value(x){
    }

    unique_ptr<Base> clone() const{
        return make_unique<Base>(value);
    }

    BigVal getValue() const{
        return value;
    }
protected:
    BigVal value;
};

struct Derived : public Base{
    Derived(BigVal x) : Base(x){
    }
};

struct Wrapper{
    unique_ptr<Base> ptr_;

    BigVal getValue() const{
        return ptr_->getValue();
    }

    // Ignore missing copy/move assignment operators
    Wrapper(const Wrapper& other){ 
        ptr_ = other.ptr_->clone();
    }

    Wrapper(Wrapper&& other){ 
        ptr_ = move(other.ptr_);
    }

    Wrapper(const Base& other){
        ptr_ = other.clone();
    }  

    Wrapper(Base&& other){
        ptr_ = make_unique<Base>(std::move(other));
    } 
};

void constFun(const Wrapper& t){
    cout<<"Const fun "<<t.getValue()<<endl;
}

int main()
{
    Base b1(1);
    Base b2(2);
    Derived d1(3);

    Wrapper w = b1; // copies

    Wrapper w2 = move(w); // moves
    Wrapper w3 = move(b1); // moves

    // No copy, just taking a reference:
    const Wrapper& w4 = w2; 
    constFun(w2);

    // Can I avoid a copy here?
    const Wrapper& w5 = b2;
    constFun(b2); 
}

I've attempted to fiddle with converting constructors of the form

Base::operator const Wrapper& (){ 
  // return 'view' on this object
} 

But that results in ambiguity when doing Wrapper w = Derived();

Another option could be to make Wrapper never make copies unless you call clone, or lazily make copies when it changes. But is there a way to make copies happen normally during assignment, but not when creating a reference?