How can I check at compile time that an ID is only used on the object instance that generated it?

Let's say that I have a class that manages "identifiers" that can be later used to look up data. For instance, if I wanted to manage some objects of class Foo, I could have:

Manager<Foo> manager;
Id<Foo> id = manager.create();

In this example, Id is just a generic identifier like a struct with some generic meta information that does not depend on the template parameter at all. The only thing the template parameter does is add some type safety by using a phantom type. The phantom type would prevent me from ever accidentally passing an Id<Foo> to a manager object of type Manager<Bar>, even though the underlying type of the Id is exactly the same struct. In contrast to a plain old int, the phantom type makes certain at compile time that I don't use Id with a Manager of the wrong type.

Now, I want to take this idea a step further. Instead of enforcing that Manager<Foo> only accepts identifiers of type Id<Foo>, I want to enforce that the specific instance of Manager<Foo> only accepts identifiers created by itself in the call to create(). This would prevent an object of class Manager<T> from accepting an identifier of Id<T> that was created by a different instance of Manager<T>. Since an identifier is only valid in the context of the manager that created it, this would detect at compile time that an ID is passed to the wrong manager instance even if the type being managed happens to be the same.

Specifically, I would like usage akin to the below sample code to throw a compiler error. It doesn't have to be exactly like this, but I don't want to require too much hoop jumping by the class user (myself).

Manager<Foo> m1;
Manager<Foo> m2;
Id<Foo> id1 = m1.create();
Id<Foo> id2 = m2.create();
Foo myFoo = m2.lookup(id1); // <-- causes compiler error. id1 used on m2.

I would prefer something that works in C++11 or earlier, but C++14 is acceptable, too.