Answer: all (m,n) with odd m and arbitrary n; or (m,n) with even m and even n.
(Solution of A. Zhuk.)
1. Let m be odd. Show that any positive n is appropriate.
Indeed, if P(x)≡x, then Q(P(x))≡Q(x) for any Q∈R[x].
If P(x)≡x, then P(x)−x is a polynomial of odd degree, hence it has a real root, say, a. Then for any n∈N consider Q(x)=(x−a)n. We have Q(P(x))=(P(x)−a)n. Note that P(x)−a=(P(x)−x+(x−a))≡(x−a). Hence (P(x)−a)n≡(x−a)n=Q(x), as we need.
2. Now, let m be even. First, show that n cannot be odd. Set, for example, P(x)=xm+x+1. Then P(x)−x>0 for any x∈R. If n is odd, then Q(x) has real roots. Let c be the largest real root of Q. That is
Q(x)=b(x−c)i=1∏k(x−ai)⋅j=1∏lqj(x),
where all qj(x) are monic quadratic polynomials with negative discriminants, c≥ai for all i=1,2,...,k, b=0.
Then
Q(P(c))=b(P(c)−c)i=1∏k(P(c)−ai)⋅j=1∏lqj(P(c)).
Note that P(c)−c>0, P(c)−ai>c−ai≥0, (∀i=1,...,k), qj(P(c))>0 (∀j=1,...,l), that is Q(P(c))=0. Hence, c is not a root of Q(P(x)), so Q(P(x))≡Q(x).
Let now both m and n be even, n=2k.
If P(x)−x has a real root a, then, as above, we set Q(x)=(x−a)n, and we are done.
Let P(x)−x has no real roots. Let z and zˉ be any complex-conjugate roots of P(x), that is P(x)−x is divisible by p(x)=(x−z)(x−zˉ)∈R[x]. Set Q(x)=(p(x))k. Then Q(P(x))=(p(P(x)))k. It suffices to prove that p(P(x)):p(x). Note that p(P(x))=p(P(x))−p(x)+p(x) and
(p(P(x))−p(x)):(P(x)−x):p(x).
Therefore,
Q(P(x))=(p(P(x)))k:(p(x))k=Q(x).