Deferred make_unique with variadic templates

I need a class which will work as a deferred factory, saving the parameters to create another class and invoking make_unique later in time. So far I'm not having any luck getting a variadic template version to work. Any help would be appreciated (minimal non-working version below).

template <typename T, typename ... Args>
class ConstructLater
{
public:
    ConstructLater(Args &&... args)
    {
        factory = std::bind(std::make_unique<T, Args...>, std::forward<Args>(args)...);
    }

    std::unique_ptr<T> Later()
    {
         return factory();
    }

private:
    std::function<std::unique_ptr<T>(void)> factory;
};

class Foo { public: Foo(int) { } };

int f()
{
    // None of these work
    ConstructLater<Foo>(3);
    ConstructLater<Foo, int>(6);
}