Answer: f(z)=0 or f(z)=z2.
Replacing y by −y gives us
f(x)+f(y)2+x2f(xy)=f(x+y2)=f(x)+f(−y)2+x2f(−xy),
which implies that
f(y)2+x2f(xy)=f(−y)2+x2f(−xy)(7)
for all x=0 and all y. Let x=1 and complete the squares:
(f(y)+1)2=f(y)2+2f(y)+1=f(−y)2+2f(−y)+1=(f(−y)+1)2.
Hence
f(y)+1=±(f(−y)+1),
so that, for any y,
either f(y)=f(−y) or f(y)+f(−y)=−2.
Suppose f(y)+f(−y)=−2 for all y=0. Equation (7) then simplifies to
f(xy)+1=x(f(y)+1),x,y=0.
Let y=1 (or swap x and y) to deduce
f(x)=(1+f(1))x−1=ax−1,x=0,
where a=1+f(1). Insert this expression into the original equation:
(a2−a)y2+1=x2,x,y,x+y2=0.
This is clearly impossible (fix one y=0 and let x=1 and x=2).
The contradiction shows that f(−y)=f(y) for some y=0. Then f(−xy)=f(xy) for all x=0 by (7), so that f(−z)=f(z) for all z.
Returning to the original equation, let y=1:
f(x+1)=(1+x2)f(x)+f(1)2,x=0.
Exchange x for −x−1 and use f(−x)=f(x):
f(x)=(1−x+12)f(x+1)+f(1)2,x=−1.
Eliminate f(x+1) from these two equations and simplify:
f(x)=x2f(1)2,x=0,−1.(8)
This formula is, in fact, accurate even for x=0 and x=−1. Indeed, letting y=0 in the original equation leads to
f(0)=0=02f(1)2.
Moreover, letting x=1 in (8) yields f(1)=f(1)2, so that
f(−1)=f(1)=(−1)2f(1)2.
Finally, f(1)=f(1)2 determines f(1)=0 or f(1)=1. Therefore, (8) provides two easily verified solutions f(z)=0 and f(z)=z2. □