Is it good to use int_fastN_t to replace intN_t

I just read this link: The difference of int8_t, int_least8_t and int_fast8_t? and now I know that int8_t is exactly 8 bits whereas int_fast8_t is the fastest int type that has at least 8 bits.

I'm a developer who develops backend processes with c++11 on Linux. Most of time I don't need to worry about the size of my processes. But I need always care about the sizes of integers in my project. For example, if I want to use an int to store the ID of user or to store a millisecond-timepoint, I can't simply use int because it may cause overflow, I must use int32_t or int64_t.

So I'm thinking if it's good to use int_fast8_t everywhere and stop using int8_t (same as int_fast32_t, int_fast64_t, uint_fast8_t etc).

Well, using int_fastN_t may change nothing because my program is always deployed on X86 or Arm64. But I still want to know if there is any drawback if I change all of intN_t into int_fastN_t. If there isn't any drawback, I think I would start to use int_fastN_t and stop using intN_t.