All polynomials satisfying the problem's conditions are:
(i) f(x)=g(x)
(ii) f(x)=S(x+a)−a,g(x)=−S(x+a)−a, where a is any real number, and S is any odd polynomial function (i.e. S(x)=∑aix2k+1).
It is easy to see that f(x) and g(x) have the same degree. Let F(x)=f(f(x))=∑i=0mamxm−i,
G(x)=g(g(x))=∑i=0mbmxm−i. Then the polynomials F(x),G(x) satisfy F(F(x))=G(G(x)). Write H(x)=F(F(x))=G(G(x))=∑i=0m2hixm2−i, and we will examine h0,h1,…,hm in order.
Expanding F(F(x)):
F(F(x))=a0(a0xm+a1xm−1+⋯+am)m+a1(a0xm+⋯+am)m−1+…
Note that only the term a0(a0xm+a1xm−1+⋯+am)m has degree in x exceeding m(m−1), so when computing h0,h1,…,hm−1, we only need to look at this term.
First, from
h0=a0m+1=b0m+1
we can deduce that a0=b0 or a0=−b0, and the latter can only happen when m is odd. For convenience of notation, we write a0=sb0, where s=1 or −1.
Next we use mathematical induction to prove that ai=sbi,∀0≤i<m. The base case i=0 has already been proven.
Suppose for some positive integer k<m, ai=sbi holds for all 0≤i<k, then observe the equation
hk=a0r1+⋯+rm=k∑ar1ar2⋯arm=b0r1+⋯+rm=k∑br1br2⋯brm
Among the terms in the Σ, if a term has every ri less than k, then every ri satisfies ari=sbri, and thus
a0ar1ar2⋯arm=sm+1b0br1br2⋯brm=b0br1br2⋯brm
Therefore this term cancels out on both sides of the equation. The only remaining terms that cannot be cancelled are those where for some i, ri=k and all others are 0. There are m such terms in total, so
ma0mak=mb0mbk=msma0mbk
so ak=sbk. By mathematical induction, we have proven ai=sbi,∀0≤i<m.
It is worth mentioning that when k=m, hk will have one additional term beyond the above, namely a1a0m−1 and b1b0m−1=sma1a0m−1 (coming from the highest degree of the second largest term of F(F(x))). This means that when s=1 we can still continue to deduce am=sbm, giving one set of solutions F(x)=G(x); but when s=−1 these extra two terms cannot cancel, so we cannot obtain information about the constant term, and we only get F(x)=−G(x)+r, where r is some real number.
We continue the discussion for the case F(x)=−G(x)+r. Substituting this into F(F(x))=G(G(x)) gives
G(G(x))=F(F(x))=−G(−G(x)+r)+r
Recall that in this case m is odd, so G(x) maps onto all real numbers, hence for any real number a we can find x such that G(x)=2r+a, so
G(2r+a)−2r=−(G(2r−a)−2r),∀a∈R
This tells us that G(x) is an odd function centered at (2r,2r).
At this point, we have deduced that if F(F(x))=G(G(x)), then F(x)=G(x), or F(x)=S(x+a)−a,G(x)=−S(x+a)−a, where a is any real number, and S(x) is any odd polynomial function. Substituting back to check, it is easy to see that both sets of solutions satisfy F(F(x))=G(G(x)).
Returning to the original problem, let the degrees of f(x) and g(x) be k, with k2=m. If we are in the case F(x)=S(x+a)−a,G(x)=−S(x+a)−a, then f(f(x))=F(x) and g(g(x))=G(x) have leading coefficients differing by a sign.
But recall that in this case m is odd, so k is also odd, and then the leading coefficient of f(f(x)) will be a0k+1>0, and similarly the leading coefficient of g(g(x)) is also greater than zero, a contradiction!
Therefore only the case F(x)=G(x) remains, in which case f(f(x))=g(g(x)), so we immediately know that all solutions satisfying the problem's conditions are:
(i) f(x)=g(x)