How do physics engine check if two objects belong to the right categories for overlap?

Take a look at this image:

Many physics engines (Box2d etc) follow such a system, where they allow you to define which category (Cat.) that an object belongs to, and the Mask - the categories of objects that it can collide with, and based on this determine if a collision should take place. I'm trying to implement the same, as I have an analogous problem, and need only this (no physics).

For example, an object (Object1) may belong to multiple categories - 0,1,2,3 and collide with categories (Mask), 5,8,9. Another Object (Object2), can belong to category (5), and collide only with Category (3). In this case a collision between Object1 and Object2 can take place.

Additional example, an object (Object1) may belong to multiple categories - 0,1,2,3 and collide with categories (Mask), 5,8,9. Another Object (Object2), can belong to category (5), and collide only with Category (4). In this case a collision between Object1 and Object2 will not take place, because although Object1 is interested in colliding with objects in category 5, Object2 is not interested in colliding with any of Object1's categories.

The collisions will take place only if A collides with B, and B collides with A.

I am trying to implement something which does the same, and am trying to figure out the data structure and algorithm to use, to efficiently check if two objects should "collide", according to the above logic. By efficiently, I mean use minimal memory and cpu required, so that I can do this check every clock tick - I am not trying to optimize specifically for memory or cpu, but simply find a solution that doesn't use more than it requires of each!

Here is the algorithm and data structures I've considered so far:

std::unordered_map<int,std::unordered_set<GameObject*>> categoryToObject
std::unordered_map<int,std::unordered_set<GameObject*>> maskToObject
std::unordered_map<GameObject*,std::unordered_set<GameObject*> objectsItCollidesWith

Algorithm:

Each time a new GameObject is created, for every category and mask it belongs to, it is added to the appropriate vector in each map. That means an object with Cat{1,3,7} & Mask{2,7} would be added to three vectors in the Cat map and two vectors in the mask map.

Just before adding to each map, for a new object, iterate through the categories of the new object, and use each category as a key in maskToObject, to see which gameObjects are interested in colliding with it. For each of these objects, check their category, and see if any of their categories fall into the mask that this object collides with. If that is the case, then add this object to a set of objects that this object collides with, and vice versa for the other object.

This way, as long as the categories / masks don't change, we just need to refer to the objectItCollidesWith map.

Is there a more efficient way than this ? Since the initial calculation needs to be done only once, and doesn't need to be done again unless the category / mask changes and I thought this was good.