From the task of the problem when substituting y=f(y) we have that
f(f(x)+f(y))f(f(x)+f(y))−f(f(x))−2f(y)f(x)−f(f(y))=f(f(x))+2f(y)f(x)−f(f(y))+2f2(y)+1⇒=−2f(f(y))+2f2(y)+1.
Therefore, due to the symmetry of the left part relative to the variables x,y the following equality must be obtained:
f(f(y))=f2(y)+c, where c=const⇒f(f(x)+f(y))=f2(x)+c+2f(y)f(x)−f2(y)−c+2f2(y)+1 andf(f(x)+f(y))=(f(x)+f(y))2+1.(1)
Let's fix some x0, and let y0=f(x0). Then if x=x0 we get that
f(f(x0)+y)+f(y)=f(f(x0))+2yf(x0)+2y2+1, orf(y0+y)+f(y)=f(y0)+2yy0+2y2+1⇔f(y0+y)+f(y)=2yy0+2y2+f(y0)+1.
Therefore for any t>f(y0)+1 exists yt such that 2yy0+2y2+f(y0)+1=t, thus f(y0+yt)+f(yt)=t. Taking into account (1), substituting x=y0+yt and y=yt in the given equation we obtain:
f(f(y0+yt)+f(yt))=(f(y0+yt)+f(yt))2+1 orf(t)=t2+1(2)
for any t>f(y0)+1=a.
For any x>a we have that x2+1≥2x>x>a, from the statement of the problem if x>a applying equality (2) several times, we get that
f(x2+1+y)=f(x2+1)+2y(x2+1)−f(y)+2y2+1 orf(x2+1+y)=(x2+1+y)2+1=(x2+1)2+1+2y(x2+1)−f(y)+2y2+1.
After opening all brackets we get that for any positive y we have equality:
f(y)=−x4−1−y2−2x2−2x2y−2y+x4+2x2+1+2yx2+2y+2y2+1⇔f(y)=y2+1.
It is straightforward to check that f(x)=x2+1 satisfies the conditions of the problem.