Let us label the equation given in the problem by (⋆). We will show that the functions f satisfying the equation (⋆) are given by the following:
f(x)=0,x2,−x2,x2−1,1−x2.
It is easy to check that all of these functions satisfy the equation (⋆).
We will show in the sequel that there are no other function which satisfies the equation (⋆).
So, let a function f satisfy the equation (⋆) and suppose for some t, f(t)=0 is true. Suppose for some pair of real numbers a,b the condition f(a)=f(b) is satisfied. By comparing the results obtained by substituting into (⋆) (x,y)=(a,t) and (x,y)=(b,t), we see that 2a2f(t)=2b2f(t) must hold, from which we get a=±b.
Let f(0)=k. Substituting (x,y)=(0,0) into the equation (⋆), we see that
f(k)=0 must be satisfied. Also, if we substitute (x,y)=(0,k) into (⋆) and use
the fact f(k)=0, we get f(k2)=0. Thus, we see that we must have k2=±k,
and therefore k=0,±1. Finally, substituting (x,y)=(0,y) into (⋆), we get
f(k−f(y))=f(y2), from which we can conclude that k−f(y)=±y2 for any y,
i.e., for any y, f(y)∈{k+y2,k−y2} must hold.
(1) Let us consider the case k=0.
Suppose there exist non-zero real numbers z,w for which f(z)=z2, f(w)=−w2 are satisfied. By substituting (x,y)=(w,w) into the equation (⋆), we obtain 0=f(f(w))+2w4+f(w2). Since k=0, we have f(f(w))=±w4 and f(w2)=±w4, and we can check that f(w2)=−w4 must hold, since w=0. Finally, if we substitute (x,y)=(z,w) into the equation (⋆), we get
±(z2+w2)2=±z4+2z2w2−w4.
By simplifying each of the 4 cases arising from the combinations of ± signs,
we can check easily that there are no (z,w)'s with z2>0, w2>0 which
satisfies the last equation. We can thus conclude that if k=0, then either
f(x)=x2 for all x or f(x)=−x2 for all x.
(2) Consider the cases when k=±1.
By substituting (x,y)=(x,k) into the equation (⋆), we get
f(f(x)−k)=f(f(x))−2kx2+k.
From f(x)−k=±x2, we have f(f(x)−k)=k±x4, and since f(f(x))=k±f(x)2 we get
(⋆⋆)±f(x)2=±x4+2kx2−k.
• Suppose k=1 holds.
If there exists a c=0 for which f(c)=1+c2, then by substituting x=c into the equation (⋆⋆) above, we get
±(1+c2)2=±c4+2c2=1.
But we can check easily that this is impossible for any combination of ± signs, since c2>0. Hence, if k=1 we must have f(x)=1−x2 for all x.
• Suppose k=−1 holds.
If there exists a c=0 for which f(c)=1−c2, we get from the equation (⋆⋆) that
±(−1−c2)2=±c4−2c2+1.
But as for the case of k=1, this identity (for any combination of ± signs) does not hold since c2>0. Thus we must have f(x)=−1+x2 for all x.
Putting together the arguments made above, we conclude that there are only 5 possibilities: 0,x2,−x2,x2−1,1−x2 for f satisfying the equation (⋆).