(1) Prove that f(1)=1.
Put y=1 in \textcircled{1}, and write f(1)=a. Then
f(x)+a+2xf(x)=f(x+1)f(x)
Thus
f(x+1)=(1+2x)f(x)+af(x)2◯
Hence
f(2)=4aa=41,
f(3)=41+a1=5+4a1,
f(4)=7+5a+4a21.
---
On the other hand, we put x=y=2 in \textcircled{1}, then
2f(2)+8f(4)=f(4)f(4)=1.
By \textcircled{2}, we have
21+7+5a+4a28=1.
Solving the equation, we have a=1, i.e. f(1)=1.
(2) Prove that
f(x+n)=(n2+2nx)f(x)+1f(x),n=1,2,…3◯
Firstly, according to \textcircled{2} we know that \textcircled{3} is true for n=1. Now suppose \textcircled{3} is true for n=k. Then
f(x+k+1)=(1+2(x+k))f(x+k)+1f(x+k)=((k2+2kx)f(x)+1f(x))/((k2+2kx)f(x)+1(1+2(x+k))f(x)+1)=((k+1)2+2(k+1)x)f(x)+1f(x).
Hence the result follows by induction.
From \textcircled{3} we have
f(n+1)=(n2+2n)f(1)+1f(1)=(n+1)21,
so f(n)=n21,n=1,2,….
(3) Prove that
f(n1)=n2=(n1)21,n=1,2,…4◯
In fact, by letting x=n1 in \textcircled{3}, we have
f(n+n1)=(n2+2)f(n1)+1f(n1),
and by setting y=x1 in \textcircled{1}, we have
f(x)+f(x1)+2=f(x+x1)1.
So
f(n)+f(n1)+2=f(n+n1)1=n2+2+f(n1)1.
Consequently, f(n)=n21 implies f(n1)=n2.
(4) Prove that if q=mn, gcd(m,n)=1, m,n∈N∗, then f(q)=q21.
For m,n∈N∗, gcd(m,n)=1, put x=n, y=m1 in \textcircled{1}, we have
f(m1)+f(n)+m2nf(mn)=f(n+m1)f(mn).
Put x=m1 in \textcircled{3}, we get
f(n+m1)=(n2+m2n)f(m1)+1f(m1)=n2+m2n+m211.
So
n21+m2+m2nf(mn)=(n+m1)2f(mn).
Now we have
f(q)=f(mn)=n2+m21n21+m2=(nm)2=q21.
Finally, it is easy to verify that f(x)=x21 satisfies the condition. So f(x)=x21 is the answer.