Degree comparison of P(Q(x)) and R(S(x)) implies that q=degQ divides degS=s. We will show that S(x)=T(Q(x)) for some polynomial T. Then P(Q(x))=R(S(x))=R(T(Q(x))), so the polynomial P(t)−R(T(t)) vanishes upon substitution t=S(x); it therefore vanishes identically, as desired.
Choose the polynomials T(x) and M(x) such that
S(x)=T(Q(x))+M(x),(∗)
where degM is minimised; if M=0, then we get the desired result. For the sake of contradiction, suppose M=0. Then q∤m=degM; otherwise, M(x)=βQ(x)m/q+M1(x), where β is some number and degM1<degM, contradicting the choice of M. In particular, 0<m<s and hence degT(Q(x))=s.
Substitute now (*) into R(S(x))−P(Q(x))=0; let α be the leading coefficient of R(x) and let r=degR(x). Expand the brackets to get a sum of powers of Q(x) and other terms including powers of M(x) as well. Amongst the latter, the unique term of highest degree is αrM(x)T(Q(x))r−1. So, for some polynomial N(x),
N(Q(x))=αrM(x)T(Q(x))r−1+a polynomial of lower degree.
This is impossible, since q divides the degree of the left-hand member, but not that of the right-hand member.
Alternative solution:
All polynomials in the solution have real coefficients. As usual, the degree of a polynomial f(x) is denoted degf(x).
Of all pairs of polynomials P(x),R(x), satisfying the conditions in the statement, choose one, say, P0(x),R0(x), so that P0(Q(x))=R0(S(x)) has a minimal (positive) degree. We will show that degR0(x)=1, say, R0(x)=αx+β for some real numbers α=0 and β, so P0(Q(x))=αS(x)+β. Hence S(x)=T(Q(x)) for some polynomial T(x).
Now, if P(x) and R(x) are polynomials satisfying P(Q(x))=R(S(x)), then P(Q(x))=R(T(Q(x))). Since Q(x) is not constant, it takes infinitely many values, so P(x) and R(T(x)) agree at infinitely many points, implying that P(x)=R(T(x)), as required.
It is therefore sufficient to solve the problem in the particular case where F(x)=P(Q(x))=R(S(x)) has a minimal degree. Let d=gcd(degQ(x),degS(x)) to write degQ(x)=ad and degS(x)=bd, where gcd(a,b)=1. Then degP(x)=bc, degR(x)=ac and degF(x)=abcd for some positive integer c. We will show that minimality of degF(x) forces c=1, so degP(x)=b, degR(x)=a and degF(x)=abd. The conditions a=degR(x) divides degP(x)=b and gcd(a,b)=1 then force a=1, as stated above.
We are now in a position to prove that c=1. Suppose, if possible, that c>1. By the lemma, there exist monic polynomials U(x) and V(x) of degree b and a, respectively, such that
deg(P(x)−U(x)c)<(c−1)banddeg(R(x)−V(x)c)<(c−1)a.
Then
deg(F(x)−U(Q(x))c)=deg(P(Q(x))−U(Q(x))c)<(c−1)abd,
deg(F(x)−V(S(x))c)=deg(R(S(x))−V(S(x))c)<(c−1)abd,
so deg(U(Q(x))c−V(S(x))c)=deg((F(x)−V(S(x))c)−(F(x)−U(Q(x))c))<(c−1)abd.
On the other hand,
U(Q(x))c−V(S(x))c=(U(Q(x))−V(S(x)))(U(Q(x))c−1+⋯+V(S(x))c−1).
By the preceding, the degree of the left-hand member is (strictly) less than (c−1)abd which is precisely the degree of the second factor in the right-hand member. This forces U(Q(x))=V(S(x)), so U(Q(x))=V(S(x)) has degree abd<abcd=degF(x) — a contradiction. Consequently, c=1. This completes the argument and concludes the proof.