It is clear that if f is a constant function which satisfies the given equation, then the constant must be 0. Conversely, f(x)=0 clearly satisfies the given equation, so, the identically 0 function is a solution. In the sequel, we consider the case where f is not a constant function.
Let t∈R and substitute (x,y,z)=(t,0,0) and (x,y,z)=(0,t,0) into the given functional equation. Then, we obtain, respectively,
f(f(t)+2f(0))=f(f(t)−f(0))+f(f(0))+2f(0),f(f(t)+2f(0))=f(f(0)−f(t))+f(f(0))+2f(0),
from which we conclude that f(f(t)−f(0))=f(f(0)−f(t)) holds for all t∈R. Now, suppose for some pair u1,u2, f(u1)=f(u2) is satisfied. Then by substituting (x,y,z)=(s,0,u1) and (x,y,z)=(s,0,u2) into the functional equation and comparing the resulting identities, we can easily conclude that
f(su1)=f(su2)(*)
holds for all s∈R. Since f is not a constant function there exists an s0 such that f(s0)−f(0)=0. If we put u1=f(s0)−f(0),u2=−u1, then f(u1)=f(u2), so we have by (∗)
f(su1)=f(su2)=f(−su1)
for all s∈R. Since u1=0, we conclude that
f(x)=f(−x)
holds for all x∈R.
Next, if f(u)=f(0) for some u=0, then by (∗), we have f(su)=f(s0)=f(0) for all s, which implies that f is a constant function, contradicting our assumption. Therefore, we must have f(s)=f(0) whenever s=0.
We will now show that if f(x)=f(y) holds, then either x=y or x=−y must hold. Suppose on the contrary that f(x0)=f(y0) holds for some pair of non-zero numbers x0,y0 for which x0=y0,x0=−y0. Since f(−y0)=f(y0), we may assume, by replacing y0 by −y0 if necessary, that x0 and y0 have the same sign. In view of (∗), we see that f(sx0)=f(sy0) holds for all s, and therefore, there exists some r>0,r=1 such that
f(x)=f(rx)
holds for all x. Replacing x by rx and y by ry in the given functional equation, we obtain
f(f(rx)+f(ry)+f(z))=f(f(rx)−f(ry))+f(2r2xy+f(z))+2f(r(x−y)z)(i)
and replacing x by r2x in the functional equation, we get
f(f(r2x)+f(y)+f(z))=f(f(r2x)−f(y))+f(2r2xy+f(z))+2f((r2x−y)z)(ii)
Since f(rx)=f(x) holds for all x∈R, we see that except for the last term on the right-hand side, all the corresponding terms appearing in the identities (i) and (ii) above are equal, and hence we conclude that
f(r(x−y)z)=f((r2x−y)z))(iii)
must hold for arbitrary choice of x,y,z∈R. For arbitrarily fixed pair u,v∈R, substitute (x,y,z)=(r2−1v−u,r2−1v−r2u,1) into the identity (iii). Then we obtain f(v)=f(ru)=f(u), since x−y=u,r2x−y=v,z=1. But this implies that the function f is a constant, contradicting our assumption. Thus we conclude that if f(x)=f(y) then either x=y or x=−y must hold.
By substituting z=0 in the functional equation, we get
f(f(x)+f(y)+f(0))=f(f(x)−f(y)+f(0))=f((f(x)−f(y))+f(2xy+f(0))+2f(0).
Changing y to −y in the identity above and using the fact that f(y)=f(−y), we see that all the terms except the second term on the right-hand side in the identity above remain the same. Thus we conclude that f(2xy+f(0))=f(−2xy+f(0)), from which we get either 2xy+f(0)=−2xy+f(0) or 2xy+f(0)=2xy−f(0) for all x,y∈R. The first of these alternatives says that 4xy=0, which is impossible if xy=0. Therefore the second alternative must be valid and we get that f(0)=0.
Finally, let us show that if f satisfies the given functional equation and is not a constant function, then f(x)=x2. Let x=y in the functional equation, then since f(0)=0, we get
f(2f(x)+f(z))=f(2x2+f(z))
from which we conclude that either 2f(x)+f(z)=2x2+f(z) or 2f(x)+f(z)=−2x2−f(z) must hold. Suppose there exists x0 for which f(x0)=x02, then from the second alternative, we see that f(z)=−f(x0)−x02 must hold for all z, which means that f must be a constant function, contrary to our assumption. Therefore, the first alternative above must hold, and we have f(x)=x2 for all x, establishing our claim.
It is easy to check that f(x)=x2 does satisfy the given functional equation, so we conclude that f(x)=0 and f(x)=x2 are the only functions that satisfy the requirement.