*Answer:* f(x)=0 and f(x)=x.
Substituting x=y=0 to the equation, we get (f(0))2=0+2f(0)+(f(0))2. Simplifying this gives f(0)=0.
Substituting y=0 to the original equation, we get the equation (f(x))2=xf(x)+2f(0)+(f(0))2 which must be satisfied for all real x. Since f(0)=0, this simplifies to f(x)(f(x)−x)=0. Thus, for each x, either f(x)=0 or f(x)=x.
Assume there exists a real number c=0 such that f(c)=0. Substituting x=c to the original equation, we get (f(c+y))2=2f(cy)+(f(y))2, or
2f(cy)=(f(c+y))2−(f(y))2.(1)
This is valid for any y. Let's analyse four cases based on whether f(c+y)=0 or f(c+y)=c+y and whether f(y)=0 or f(y)=y.
1) If f(c+y)=0 and f(y)=0, then (1) simplifies to f(cy)=0.
2) If f(c+y)=0 and f(y)=y, then (1) gives f(cy)=−2y2. Assuming that f(cy)=cy=0, we get c=−2y i.e. y=−2c.
3) If f(c+y)=c+y and f(y)=0, then (1) gives us 2f(cy)=(c+y)2. Assuming that f(cy)=cy, we get c2+y2=0 from which c=0, contradiction.
4) If f(c+y)=c+y and f(y)=y, then (1) gives us 2f(cy)=c2+2cy. Assuming that f(cy)=cy, we get c2=0, contradiction.
Thus f(cy)=0 can only be valid if y=−2c, i.e. f(x)=0 can only be valid if x=c⋅(−2c)=−2c2. Thus it is possible to choose a real number d such that d=0, f(d)=0, and ∣d∣=∣c∣. By replacing d by c we can analogously conclude that f(x)=0 can only be valid when x=−2d2. As −2d2=−2c2, f(x)=0 cannot be valid for any x=0. So f(x)=0 for all x.
Therefore the only suitable functions are f(x)=0 and f(x)=x.