Solution:
Set α=f(1). Then setting y=1 and x=1 in
f(x2+y)=f2(x)+f(x)f(xy)
gives
f(x2+1)=f2(x)+1
and
f(y+1)=α2+αf(y)
respectively. Using (3), we consecutively get
f(2)=α2+1, f(3)=αα3+α2+1f(4)=α2α4+α3+α2+1, f(5)=α3α5+α4+α3+α2+1
On the other hand, setting x=2 in (2) gives f(5)=α4+2α2+2. Therefore α3α5+α4+α3+α2+1=α4+2α2+2⟺α7+α5−α4+α3−α2−1=0, whence
(α−1)[α4(α2+α+1)+(α+1)2(α2−α+1)+2α2]=0
Since the expression in the square brackets is positive, we have α=1. Now (3) implies that
f(y+1)=f(y)+1
and therefore f(n)=n for every positive integer n.
Now take an arbitrary positive rational number ba ( a,b are positive integers). Since (4) gives f(y)=y⟺f(y+m)=y+m,m is a positive integer, the equality f(ba)=ba is equivalent to
f(b2+ba)=b2+ba
Since the last equality follows from (1) for x=b and y=ba we conclude that f(ba)=ba.
Setting y=x2 in (4), we obtain f(x2+1)=f(x2)+1. Hence using (2) we conclude that f(x2)=f2(x)>0. Thus f(x)>0 for every x>0. Now (1), the inequality f(x)>0 for x>0 and the identity f(x2)=f2(x) imply that f(x)>f(y) for x>y>0. Since f(x)=x for every rational number x>0, it easily follows that f(x)=x for every real number x>0.
Finally, given an x<0 we choose y<0 such that x2+y>0. Then xy>0 and (1) gives
x2+y=f(x2+y)=f2(x)+f(x)f(xy)=f(x2)+f(x)xy=x2+f(x)xy
i.e. f(x)=x. Therefore f(x)=x for every x∈R∗. It is clear that this function satisfies (1).