Custom allocator without pointer arithmetic

I have a custom allocator class which looks something like this:

class allocator {
private:
  char* memory;
  std::ptrdiff_t offset;
  std::size_t total_size;

public:
  allocator(std::size_t size) : total_size(size), offset(0) {
    memory = new char[size];
  }
  void* allocate(std::size_t size, std::size_t alignment) {
    // unimportant stuff

    void* current_address = &start_ptr[offset]; // --> OUCH, pointer arithmethic

    offset += size;
    // unimportant stuff
    return current_address;
  }

}

As you can see above I'm using pointer arithmetic to calculate the current address for the newly allocated block of memory. The CppCoreGuidelines and many other guidelines discourage the use of pointer arithmethic. So is there another way of managing the memory pool?

I was thinking of maybe using a std::vector<char>, as it contains a contiguous block of memory, and doing something like this:

std::vector<char> memory;

void* current_address = &(memory.at(offset));

But this doesn't seem any nicer to me. Do you have any ideas of how to cleanly manage the memory pool in a safe way?