OpenCV mouse Listener doesn't get called

I created an OpenCV program in which i want:
1. To display 5 pictures
2. When left clicked on one of the pictures additional 36 pictures get displayed/ When right clicked on the parent picture (one of 5 pictures) those images get destroyed(hidden)
3. When clicked on one of those 36 pictures additional 2 pic get shown and again left click shows right click hides.
4. When pressed 'q' all images are closed.

I have created a special class so that i can have some sort of parent - child relationship

class DisplayNode
{
    friend RMDisplayManager;
    std::vector<DisplayNode*> children;
    cv::Mat img_to_display;
    std::string win_name;
public:
    void destroyNode();
    void createNodeWindowDisplay();
    void showNode();
    void setImg(const cv::Mat &img);
    DisplayNode(std::string win_name);
    void addChildNode(DisplayNode* child);
    void registerListenerForCurrentChildren();
    void displayChildrenNodes();
};

A short summary when displaying and creating relationships

for (unsigned iframe=0;iframe < 5; iframe++)
    {
      DisplayNode *node_img=new DisplayNode(somename+std::to_string(iframe));
      node_img->setImg(someimg);
      for (i=0;i<36;i++)
      {
            DisplayNode *node_block=new DisplayNode(somename+std::to_string(i));
            node_block->setImg(someimg);
            node_img->addChildNode(node_block);
            for (j=0;j<2;j++)
            {
                // DisplayNode's set their images 
                // and set node_block as their parent
            }
            node_block->registerListenerForCurrentChildren();
      }
      node_img->registerListenerForCurrentChildren();
  }
  waitUntilExit('q')

The mouse listener registration is defined through:

void DisplayNode::registerListenerForCurrentChildren()
{
    cv::setMouseCallback(win_name,callbackParentListener, this);
}

void callbackParentListener(int event, int /*x*/, int /*y*/, int /*flags*/, void* userdata)
{
    DisplayNode* parent_node= static_cast<DisplayNode*>(userdata);
    if  ( event == cv::EVENT_LBUTTONDOWN )
    {
        parent_node->displayChildrenNodes();
    }
    else if (event == cv::EVENT_RBUTTONDOWN)
    {
        parent_node->destroyNode();
    }
}

And function waitUntilExit wait's for char before exiting all img's

void RMDisplayManager::waitUntilExit(char exit_letter)
{
    bool exit=false;
    while (!exit)
    {
        char character_press=cv::waitKey(30);
        if (character_press==exit_letter)
        {
            exit=true;
            destroyAllNodes();
        }

    }
}

However the program just displays the 5 images correctly and doesn't respond at all when i click on any of the images.
callbackParentListener doesn't get called (i checked in gdb).
I'm assuming that waitUntilExit goes into infinite loop but i'm not sure. Still, how can one have a keyboard and a mouse listener in OpenCV that work together (if waitKey(30) releases the CPU mouse listener should be able to function correctly i guess? )