Is it possible for the destructor to NOT call the destructor of a specific field?

I am using a C++ compiler with the -std=c++0x flag.

I am writing a simulation for a virtual memory system for class and have ran into a problem:

One of the classes, VirtualMemory, initializes several objects of a different class called OurPointer by giving them an integer and a pointer to itself.

The program runs several pointers that are all linked to the same virtual memory. the constructor of OurPointer looks like so:

OurPointer::OurPointer(int adr , VirtualMemory* vrtlMem)
{
    _adr = adr;
    _vrtlMem = vrtlMem;
}

This is because OurPointer is used to work with VirtualMemory. The problem is, every time an OurPointer object is destroyed, it calls the destructor of _vrtlMem, which points to the same VirtualMemory object that is still used by the program.

Is there a way to exclude that field from the destructor? I tried to turn it into a static field, or change the pointer but it didn't stop the destructor.