Move large member object out of class?

Background/Example:

I currently have a class like the following:

class Element {
  Large l1;
  OtherLarge l2;
  Small s1;
  VerySmall s2;
};

where Large and OtherLarge are relatively big (~80 bytes) whereas Small and VerySmall are fairly small (~4 to 16 Byte).

On these elements, I operate in two ways:

• sorting them in various ways. During this, only members s1 and s2 are accessed/needed.
• combining the large members in various way (e.g. matrix-matrix multiplication).

The second class of operations is already fairly fast and can be parallelised easily, hence I’d like to speed up the first class of operations more. Comparing with another class Element2 where I replaced the two big data members by 8-byte integers doing nothing tells me that if I can somehow replace the direct data members l1 and l2 by pointers of one sort or another to dynamically-allocated elements elsewhere, I’ll already get a big win.

For reference, all member types have both copy and move constructors and can be both copied and moved, but moving them is much much cheaper. Large and OtherLarge also allocate a lot of memory by themselves, so allocating a bit more isn’t necessarily horrible.

Concrete question

Is it possible, and if so, what is the best way, to replace a direct member object of a class with a pointer to a dynamically-allocated object elsewhere; preserving the behaviour of a direct member as closely as possible w.r.t construction, destruction, member variable access etc? If I use a std::unique_ptr<Large> naively, I assume I’ll have to dereference it half the time/take care of copying specially? Ideally I’d like the new member object to behave just as if the old, big member object was still there.