Cast pointer to reference pointer

In the code below I am casting pointer type to reference pointer type in order to change the pointer to point somewhere else:

inline void SWAP_POINTERS(void*& p1,void*& p2)
{
    void* temp = p1;
    p1 = p2;
    p2 = temp;
}

When I call the function I ge the following error:

note: candidate function not viable: no known conversion from 'int32_t *' (aka 'int *') to 'void *&' for 1st argument

Do I need to cast? Why do I need to cast since it is void*?

Limitation: no templates