Let us denote the equation given in the statement of a problem by (∗) and substitute y=−x into it. We obtain
f2(x)+f2(−x)+2f(−x2)=f2(0){ (**).}
Substituting x=x+y,y=−x into (∗), we get
f2(y)=f2(x+y)+f2(−x)+2f(−x2−xy).
Adding (∗) and (∗∗) to the last-mentioned equality, we obtain
f(xy)+f(−x2−xy)=f(−x2)−f2(0)/2.
Let us denote g(x)=f(x)+f2(0)/2. Let a=−x2≤0,b=xy∈R when a=0. The last-mentioned equality assumes the form
g(a)+g(b)=g(a+b){ (***)}.
If a,b<0, then we obtain the Cauchy's equation. Let us prove that g(x) is bounded above when x<0. Then from the Cauchy's equation we get that g(x)=kx for arbitrary negative number x and real number k. From (∗∗) we get
Thus, g(x)=kx∀x<0. Let us assume that x≥0. Substituting y=−1−x into (∗∗∗), we obtain g(x)=g(−1)−g(−1−x)=kx for every positive number x. Thus, we proved that
f(x)=g(x)−f2(0)/2=kx−f2(0)/2=kx+c
for every real x. Substituting the explicit form of the function f into (∗), we obtain:
k2(x+y)2+2kc(x+y)+c2(k2−k)xy=k2x2+2kcx+c2+2kxy+2c+k2y2+2kcy+c2,=c2+2c.
Considering that xy is able to possess arbitrary real value, from the last-mentioned equality we get that k∈{0,1}, c∈{0,−2}.
Thus, we obtain that the condition can be fulfilled only by the functions f(x)=0, f(x)=−2, f(x)=x, and f(x)=x−2. We verify that all these functions fulfill the condition of the problem by checking.