c++ map of pointer to structure that has mutex

I need to hold std::mutex for every object that I create to use later.

I am aware of that std::mutex is non-copyable non-movable.

Also I have looked at this:

map-of-mutex-c11

What I do is, create a std::map with objectID(key) and objectInfo* , and have the mutex in objectInfo.

So, as what I store in the map is a pointer to some structure, I do not see any problem about std::mutex, which would have occured if I had, let say, std::map<int, std::mutex>.

However, it seems that the code I have written has some problem. After inserting the code with mutexes, I get sometimes segfault when I do std::unique_lock::try_lock.

Here is minimized version of the code that I inserted in huge project:

// Example program
#include <iostream>
#include <string>
#include <mutex>
#include <map>
#include <thread>

struct sObjectInfo {
    std::mutex mut;
};

std::map<int, sObjectInfo*> mymap;

void foo() {
    std::unique_lock<std::mutex> ulock(mymap[0]->mut, std::defer_lock);
    if (ulock.try_lock()) {
        std::cout << "locked.." << std::endl;
    }
    else {
        std::cout << "cant lock" << std::endl;
    }
}

int main()
{
  sObjectInfo* s1 = new sObjectInfo();
  sObjectInfo* s2 = new sObjectInfo();

  mymap[0] = s1;
  mymap[1] = s2;

  std::thread t1(foo);
  t1.join();
}

Here is the live example:

ideone

What is wrong with the use of mutexes in this example?