Can this case of diamond inheritance be avoid, and how?

In my lib, I want export two interfaces called IB and ID. ID should be an IB, because I want use ID like an IB externally. Besides, I have two corresponding implementations, ImplB and ImplD, and ImplD is an ImplB too for the goal of using ImplD like an ImplB internally.

class IB{/*...*/};
class ID : public IB{/*...*/};
class ImplB : public IB{/*...*/};
class ImplD : public ID, public ImplB{/*...*/};

The code above will cause diamond inheritance problem.

   IB
  /  \
ImplB ID
  \  /
  ImplD

If I trying to avoid this by using virtual inheritance

class ImplB : virtual public IB{}; 
class ID : virtual public IB{};

, then another potential performance issue (additional penalty for casting to IB from ID or ImplB) will arise.

So the question is:

Is this the case where the cost above can't be get rid of?

Or there is another way to bypass all of these above, and how?