Denote by E(x,y) the expression in the problem. We have
E(x,y)=x(x2+y2)−y(x2+y2)(x+y)2=(x−y)(x2+y2)(x+y)2(1)
and we can assume that x>y.
If x−y=1, then E(x,y)∈Z if and only if x2+y2∣2xy. This is not possible because x2+y2>2xy.
If x−y>1, then x−y≥2. We have
0<E(x,y)≤2(x2+y2)(x+y)2<1
hence E(x,y)∈/Z.