Answer: P(x)=ax2−a(b+1)x+ab, Q(x)=x2−(b+1)x+2b, where a,b∈R and a=0.
Denote the degrees of polynomials P and Q by m and n, respectively. In the equality
P(Q(x))=P(x)Q(x)−P(x)(1)
compare the degrees of both sides:
mn=m+n⟺(m−1)(n−1)=1⟺m=n=2.
Therefore we can denote P(x)=αx2+βx+c. Hence (1) can be written in the form α(Q(x))2+βQ(x)=P(x)Q(x)−(P(x)+c)Q(x). Since polynomials P and Q have equal degrees, P(x)+c=aQ(x) for some real number a=0. The equality (1) now takes the form
aQ(Q(x))−c=(aQ(x)−c)Q(x)−(aQ(x)−c)=a(Q(x))2−(a+c)Q(x)+c.
Since Q is nonconstant, it attends infinitely many values, hence aQ(x)=ax2−(a+c)x+2c, or equivalently Q(x)=x2−(b+1)x+2b (where b=ac). Then P(x)=aQ(x)−ab, i.e. P(x)=ax2−a(b+1)x+ab. It is easy to verify that such polynomials P and Q satisfy the problem conditions (1). Indeed, since P and Q satisfy P(x)=aQ(x)−ab, the equality (1) can be written as aQ(Q(x))−ab=(aQ(x)−ab)Q(x)−(aQ(x)−ab), or, after simplification, Q(Q(x))=(Q(x))2−(b+1)Q(x)+2b, which is true for Q(x)=x2−(b+1)x+2b.