C++11: template specialization to no-action

I have a class X which looks as follows:

template <typename T>
class X {
 public:
   X(T &t): t_ {t} {}

 private:
   T &t_;
};

Is there a way to specialize it for T = int such that an expression X<int>(i) for an integer i does nothing but "returns" just i compile time?

In other words, I would like to have the same syntax for double and int (X<double>(d) and X<int>(i) respectively) but the former calls a constructor that initializes a reference as showed in the code above, while the latter ignores that "syntax sugar" and "returns" integer i without doing anything.