We will show the claim by assuming the contrary.
Assume that there exist non-constant polynomials G(x,y) and H(x,y), with real coefficients, such that
P(x,y)=G(x,y)⋅H(x,y),(1)
where P(x,y)=xn+xy+yn, n∈N∗.
Present G(x,y) and H(x,y) in the form of polynomials in x:
G(x,y)H(x,y)=gm(y)⋅xm+gm−1(y)⋅xm−1+⋯+g1(y)⋅x+g0(y),m∈N;=hk(y)⋅xk+hk−1(y)⋅xk−1+⋯+h1(y)⋅x+h0(y),k∈N;
where gi(y), i=0,…,m, and hj(y), j=0,…,k, are real polynomials in y.
It follows from (1):
m+k=n,(2)
For n≥2, gm(y),hk(y) are constant polynomials and hence are not divisible by y.(3)
Since G(x,y) and H(x,y) are non-constant, by means of (3), if n≥2, then m,k≥1.(4)
* If n=1. Then, according to (2), we have m+k=1. Consequently, m=0 and k=1, or m=1 and k=0.
Assume that m=0 and k=1. (The case m=1 and k=0 is treated similarly). Then we have
(y+1)x+y=g0(y)h1(y)x+g0(y)h0(y).
Consequently g0(y)(h1(y)−h0(y))=1. Thus, g0(y) is a constant polynomial, contradicting the assumption that G(x,y) is non-constant.
* If n≥2.
Let i0 and j0 be the least indices such that gi0(y) and hj0(y) are polynomials not divisible by y.
Clearly, the coefficients of xi0+j0 in the expansion of G(x,y)H(x,y) are
g0(y)hi0+j0(y)+g1(y)hi0+j0−1(y)+⋯+gi0(y)hj0(y)+gi0+1(y)hj0−1(y)+⋯+gi0+j0(y)h0(y)
It follows from the definition of i0 and j0 that the above coefficients are not divisible by y. Thus, by (1), with the remark that the coefficient of xn in P is the unique one not divisible by y, we conclude i0+j0=n. Hence i0=m and j0=k. Together with (4) we have either m=1 or k=1, as if otherwise, m,k>1, by comparing the coefficients of x on both sides of (1) we would have y=g0(y)h1(y)+g1(y)h0(y)=y2, a contradiction.
Assume m=1. (The case k=1 is treated similarly). Then we have
xn+xy+yn=(ax+g0(y))(bxn−1+hn−2(y)xn−2+⋯+h1(y)x+h0(y)),(5)
where a,b are real constants with b=1.
By (5) we have yn=g0(y)h0(y). Consequently g0(y)=a′ys, where s∈N∗,s≤n and a′ is a real constant, different from 0.
Put c=−aa′, we have c=0. Plug x=cys in (5), we obtain
cnysn+cys+1+yn=0.(6)
+ If s=1 and n=2, we obtain from (6): (c2+c+1)y2=0. Consequently c2+c+1=0, a contradiction.
+ If s=1 and n>2, we obtain from (6): (cn+1)yn+cy2=0, a contradiction (since c=0).
+ If s≥2 and n≥2 then sn>n and sn>s+1. Hence (6) is contradictory, since c=0.
* Thus, in conclusion, the assumption at the very beginning is wrong and we thereby verify the claim of the problem.