Make object keep pointer to other objects - but not add code to the object directly (decoupling)

I have an entity-based-component system.

A GameObject (entity) keeps all pointers to all related GameComponents.
All GameComponent keep a pointer to the GameObject.

If I know that I have to query one component from another component often (Physics<->Graphics), I will cache pointer of another component.

class Physics : public GameComponent {
    Graphics* graphic;  //different GameObject
};
class Graphics: public GameComponent {
    Physics * physic ;  //different GameObject
};

The cost of querying pointer from each other (Physics->Graphics and Graphics->Physics) is very small.

Version 2

As time go by, storing Graphics's pointer in Physics and vice versa becomes less sensible and causes maintainability problem.

For example, if I want to add a new relation, e.g. Physics-Player, I will have to add a field into both classes :-

• add Player* to class Physics
• add Physic* to class Player

Finally, totally simple components will be clogged with many fields about the relation.

class Physic: public GameComponent {
    Graphic* graphicsMain;
    Graphic* graphicsSecondary;
    Player* playerPtr;
    Enemy* enemyPtr;
    //10+ as project grow
}

Code about query an object from another object are scattering around in many places.

Question: How to solve the maintainability issue without performance penalty?

My workaround

I created a new GameComponent named RelationNode.
Then create a new system named Relation to manage every type of relation in a single place.
Pros: I can code all utility that I want, and it will work for every relation type.

class Physics : public GameComponent {
    //no keep Graphics pointer anymore ... clean :)
};
class Graphics: public GameComponent {
    //no keep  Physics pointer anymore ... clean :)
};
public RelationNode : public GameComponent {
    //... some code ... cache other component ...
    RelationNode* slot01; //shared Physics->Graphics and Graphics->Physics
    RelationNode* slot02; //Physic->graphicsSecondary
    .......
    RelationNode* slot20;
    //^ I can reuse same slot of variable if I am sure.
    //instead of "RelationNode*" , it can also be "GameObject*"
};
class Relation{ 
    Graphics* getGraphicsFromPhysics(Physics* gameComponent){
        RelationNode* a = toNode(gameComponent); //waste 1-2 indirection
        RelationNode* b = queryGraphicFromPhysics(gameComponent);
        Graphics* bCom= fromNode(b);             //waste 1-2 indirection
        return bCom;
    }
    //the real code is more elegant than this, but suffer the same issue
};

Now my code is so clean but suffer performance penalty from indirection.
It cost about 1-30% (profiled), depends on test case.

I have another crazy idea about add field to component using macro, but I don't like macro - it tends to cause other problems.