Let x=y=0, we get f2(0)=1.
If f(0)=1, let y=0, we get f(x)=1.
Substituting into equation (1) yields
1+1×1=1+2xy+1⇒xy=0.
This does not meet the problem's requirements.
Therefore, f(0)=−1.
Let x=1,y=−1, we get
f(1)f(−1)=f(−1).
(1) If f(1)=1, let y=1, we get
f(x+1)=2x+1⇒f(x)=2x−1.
(2) If f(−1)=0, let x=−2,y=1, we get
f(−2)f(1)=f(−2)−3;
Let x=y=−1, we get f(−2)=f(1)+3. Substituting into the right side of equation (2) yields f(−2)f(1)=f(1).
(i) f(1)=0.
Substitute x−1 for x, let y=1, we get
f(x)=f(x−1)+2(x−1)+1;
Substitute −x+1 for x, let y=−1, we get
f(−x)=f(x−1)+2(x−1)+1.
Thus, f(x)=f(−x).
Let y=−x, we get
f2(x)=f(x2)−2x2+2;
Let y=x, we get
f(2x)+f2(x)=f(x2)+2x2+1.
Thus, f(2x)=4x2−1⇒f(x)=x2−1.
(ii) f(−2)=1, i.e., f(1)=−2.
Let y=1, we get
f(x+1)=3f(x)+2x+1.
Substitute −x for x, let y=−1, we get
f(−x−1)=f(x)+2x+1. Hence f(x+1)−f(−x−1)=2f(x)⇒f(x)−f(−x)=2f(x−1)⇒f(−x)−f(x)=2f(−x−1)⇒f(x−1)=−f(−x−1).
Thus, −f(x−1)=f(x)+2x+1.
Also f(x)=3f(x−1)+2(x−1)+1⇒f(x)=−3(f(x)+2x+1)+2x−1⇒f(x)=−x−1.
Since f(x)=2x−1,
f(x)=−x−1,f(x)=x2−1
all satisfy the conditions, hence the solutions are the above three functional equations.