Smart pointer to ofstream or cout

I'm making a simple logging class with a pointer to either a std::ofstream or std::cerr. Is there any simple way to use a smart pointer for auto clean-up regardless of which stream is used? The code must compile on clang++, g++, and VS2013.

Code

#include <iostream>
#include <fstream>
#include <string>

class Logger {
private:
    std::ostream * output_stream{ nullptr };
    bool using_file{ false };
public:
    Logger()
    {
        output_stream = &std::cerr;
        using_file = false;
    }
    Logger(std::string file)
    {
        output_stream = new std::ofstream(file);
        using_file = true;
    }
    ~Logger()
    {
        if (using_file)
        {
            delete output_stream;
        }
    }
    template<typename T>
    void log(T info)
    {
        *output_stream << info << std::endl;
    }
};

class tmp {
    int i{ 4 };
    friend std::ostream & operator<<(std::ostream &os, const tmp& p);
};

std::ostream &operator<<(std::ostream &os, const tmp& p)
{
    return os << p.i;
}

int main()
{
    tmp t;
    Logger logger;
    logger.log(t);
    system("pause");
    return 0;
}

Attempts

std::unique_ptr

I can use std::unique_ptr for the file like so:

std::unique_ptr<std::ostream> p;
p = std::make_unique<std::ofstream>("file.txt");
*p << "hi there" << std::endl;

Trying this with std::cout warns me about a deleted function (assuming that's the constructor.

std::unique_ptr<std::ostream> p2;
p2 = std::make_unique<std::ostream>(std::cout);
*p2 << "hey" << std::endl;

std::shared_ptr

Because std::unique_ptr is only for owning things, and std::cout shouldn't be owned, I thought I'd try std::shared_ptr

std::shared_ptr<std::ostream> p;
p = std::make_shared<std::ostream>(std::cout);
*p << "hola" << std::endl;

It gives me the same deleted constructor error. p = &std::cout complains about a type mismatch, so it's also not working.