When should I use the C++ fixed-width integer types and how do they impact performance?

Normally I imagine most people would use int for everything and ocassionally they would use unsigned int when needed. Every now and then you might use a short int, maybe for network traffic or something.

But lately I have started to use std::size_t for indexing into STL containers (as I should), and then I started to find a use for std::uint8_t when creating a struct of four 8-bit colour values (RGBA), rather than a char, simply because it makes more sense to me (it's a number, not a character type, and not a 8-bit value, it's a 8-bit number from 0-255).

Then while doing some network programming I found that I wanted to make sure certain values being passed were 16-bits. And according to https://en.cppreference.com/w/cpp/language/types I can't assume a short int is 16 bits, only that it's at least 16 bits. So I found a use for std::int16_t.

This led me to gradually start using a fixed-width type everywhere. Which caused me to really think about what I needed, how much range I needed etc, whether it would fall into negative values or not.

So now, I have almost zero occurences of int in my code.

There's three reasons in my head for this:

1. It makes my intent clear to someone else (i.e., I don't expect the number to be larger than this or it won't be negative)
2. Code is more portable. If I want a 16bit integer type, it will be 16 bit on every compiler
3. Saves memory. Though I imagine on a modern PC this is largely unimportant.

But I'm concerned that this is reducing performance as your usual consumer-grade CPU would operate on a native width type like a std::int32_t better.

So, when should I use the fixed-width types, and how do they impact performance?