std::shared_ptr with joined control block and custom deleter

I'm trying to create a std::shared_ptr with a custom deleter where the allocation of the control block is joined with the one of the object itself.

The constructor of std::shared_ptr which accepts a custom deleter can't be used since this causes another allocation for the control block itself:

std::shared_ptr<int> ptr(new int(0), [] (int*) {
  // ...
});

Further I came over std::allocate_shared such that I accomplished to provide a custom allocator for the control block and object:

template <typename Object>
class my_allocator {
public:
  using value_type = Object;

  template <typename O>
  explicit my_allocator(my_allocator<O> const& other) {
  }

  Object* allocate(std::size_t n, const void* hint = nullptr) {
    assert(n == 1U);
    return static_cast<value_type*>(::operator new(sizeof(value_type)));
  }

  void deallocate(Object* p, std::size_t n) {
    assert(n == 1U);

    // My problem occurs here, the allocator is rebound to
    // an internal type provided by the standard library and
    // the actual pointer to the int can't be retrieved anymore.
    // The pointer can't be stored inside the allocator because it is
    // rebound to some internal type before allocation.
  }
};

// ...

using Object = int;

using allocator_t = my_allocator<Object>;
auto object = std::allocate_shared<Object>(allocator_t{this},
                                           std::forward<Args>(args)...);

The issue here is that it's only possible to control the allocation and deallocation of the joined control block but it's not possible to provide a custom deleter which I want to use to delay the actual deletion of the object.

Is it possible to use a custom deleter but with only one joined allocation?