The unique solution is f(x)=x−1,x∈R.
We let g(x)=f(x)+1 and prove that g(x)=x for all real numbers x. Then the condition of the problem can be written as
g(1+xy)−g(x+y)=(g(x)−1)(g(y)−1)for all x,y∈R with g(−1)=1.(1)
Let C=g(−1)−1=0. Setting y=−1 in (1), we obtain
g(1−x)−g(x−1)=C(g(x)−1).(2)
Setting x=1 in (2) gives C(g(1)−1)=0. Therefore, since C=0, we have g(1)=1. Substituting x=0 and x=2 respectively produces g(0)=0 and g(2)=2.
We observe that
g(x)+g(2−x)=2for all x∈R,(3)
g(x+2)−g(x)=2for all x∈R.(4)
Replacing x in (2) by 1−x gives g(x)−g(−x)=C(g(1−x)−1), and replacing x by −x gives g(−x)−g(x)=C(g(1+x)−1). Adding the two equations gives C(g(1−x)+g(1+x)−2)=0. Hence, since C=0, we obtain (3).
Let u,v be such that u+v=1. Substituting (x,y) respectively by (u,v) and (2−u,2−v) in (1) gives g(1+uv)−g(1)=(g(u)−1)(g(v)−1), g(3+uv)−g(3)=
g(uv+3)−g(uv+1)=g(3)−g(1).
Every x≤45 can be expressed in the form x=uv+1 where u+v=1 (since the quadratic polynomial t2−t+(x−1) has real roots when x≤45). Therefore, when x≤45, g(x+2)−g(x)=g(3)−g(1). Since g(x)=x holds at x=0,1,2, letting x=0 gives g(3)=3. This proves that (4) holds when x≤45. If x>45 then −x<45, so g(2−x)−g(−x)=2. On the other hand, from (3) we get g(x)=2−g(2−x), g(x+2)=2−g(−x), so that g(x+2)−g(x)=g(2−x)−g(−x)=2. Hence (4) holds for all real numbers x.
Replacing x in (3) by −x gives g(−x)+g(2+x)=2. From (4) we obtain g(x)+g(−x)=0 for all x. Substituting (x,y) respectively by (−x,y) and (x,−y) in (1) gives g(1−xy)−g(−x+y)=(g(x)+1)(1−g(y)), g(1−xy)−g(x−y)=(1−g(x))(g(y)+1). Adding the two equations gives g(1−xy)=1−g(x)g(y). From (3) we obtain g(1+xy)=1+g(x)g(y). Therefore, from the original equation (1), we obtain the form g(x+y)=g(x)+g(y), so g is additive.
From additivity we get g(1+xy)=g(1)+g(xy)=1+g(xy), and since g(1+xy)=1+g(x)g(y) has already been obtained above, we get g(xy)=g(x)g(y). In particular, setting y=x gives g(x2)=g(x)2≥0 for all real numbers x, meaning that g(x)≥0 when x≥0. Since g is additive and bounded below on [0,+∞], g is linear, and moreover g(x)=g(1)x=x for all real numbers x.
In summary, the unique solution is f(x)=x−1,x∈R.