How to make wait-notify-one faster or is there an alternative way to implement wait-notify function?

In my c++ program, I have an object X, when one thread wants to handle X, it must wait for the 'X->state' being set. During this thread is waiting (by call wait()), aother thread can modify 'X->state'. When one thread modifies 'X->state', it calls notify_one(). In this case, I find that the time from one thread modifies 'X->state' and another thread awake from waiting is very high, about 20-50us (depends on how many threads running on the machine). It means that when one thread modifies the state and notifies, another thread still needs a long time to awake from waiting. Is there any way to reduce this time, or is there any other ways to implement the same functions.