Why do I get Wsign-converion warning?

I have following code:

template <typename T>
struct wrapper
{
    T t;

    operator T()
    {
        return t;   
    }

    T get()
    {
        return t;
    }
};

...

int a;
int* x = &a;
wrapper<long unsigned int> y{2};

std::cout << (x + y); // warning

When I compile it on gcc (tested on 7.3.0 and 8.2.0) with -Wsign-conversion I get "warning: conversion to 'long int' from 'long unsigned int' may change the sign of the result". If y has type long unsigned int, there is no warning. Moreover when I explicitly call y.get() there is also no warning:

std::cout << (x + y.get()); // this is ok

Why this is the case? Are there some special rules for pointer arithmetic which cannot be used when using user-defined conversion?