Will changing a private derived class affect the ABI if I use only pointers to the base class?

I've been recently reading about the PIMPL idiom in C++ and its advantages over abstract base classes. One of the main points for the PIMPL is that it preserves ABI if the implementation changes. I've seen many articles in the internet and even similar questions here in StackOverflow. However, I haven't been able to find the answer to the following question:

Let's say I have a base class A and an implementation(derived class) B. Now, external code receives B objects but only via pointers to the base class A(and they can only use whatever methods or data members are available in A). Now if I change the implementation of B, for example adding new private members, will this result in an ABI breakage? My logic tells me this shouldn't be the case, because the external code doesn't know anything about B and so changes in it shouldn't affect the external code at all.

Can someone tell me if I'm wrong and also explain why? Thank you in advance.