Mixing double dispatch and static polymorphism

I'm sure this is a bad idea. Let's pretend I have a good reason to do it. I have a tree of nodes that successfully uses static polymorphism to pass messages. Crucially, each node cannot the types of the nodes it connects to, it just knows the types of messages it passes. To traverse the tree, I've implemented the visitor pattern using CRTP. This works for the first level of the tree.

However, when traversing to the second layer of the tree, the next node's type is erased using the below AnyNode class. I've been unable to figure out how to downcast from the erased type to the concrete type. The below example works in the tests, but I think it's also probably really dangerous and just working by the luck of where memory happens to be laid out.

It seems problematic that I have to erase the type of the visitor in AnyNode::Model<T>::acceptDispatch, which is fully known in AnyNode::Concept::accept. But I can't figure out how to downcast from the Concept to the Model in the Concept (I tried a covariant virtual cast function, but that didn't work). And I can't pass the typed Visitor to the derived Model class using a virtual method, because virtual methods can't be templated.

Is there a safe way to call node.accept and pass the visitor without having to erase the visitor's type and then static cast it back? Is there some way to downcast the Concept to a Model<T> at runtime? Is there a better way to approach this problem? Isn't there some crazy new C++11 way of solving this, possibly with SFINAE?

class AnyNode
{
    struct Concept
    {
        virtual ~Concept() = default;

        template< typename V >
        void accept( V & visitor )
        {
            acceptDispatch( &visitor );
        }

        virtual void acceptDispatch( VisitorBase * ) = 0;
    };

    template< typename T >
    struct Model : public Concept
    {
        Model( T &n ) : node( n ) {}

        void acceptDispatch( VisitorBase * v ) override
        {
            // dynamic cast doesn't work, probably for good reason
            NodeVisitor< T >* visitor = static_cast< NodeVisitor< T >* >( v );
            std::cout << "CAST" << std::endl;
            if ( visitor ) {
                std::cout << "WAHOO" << std::endl;
                node.accept( *visitor );
            }
        }

    private:
        T &node;
    };

    std::unique_ptr< Concept > mConcept;
public:

    template< typename T >
    AnyNode( T &node ) :
            mConcept( new Model< T >( node )) {}


    template< typename V >
    void accept( V & visitor )
    {
        mConcept->accept( visitor );
    }
};