If f is constant, the left side is always the same, while the right side can vary, a contradiction. Therefore, f is not constant. Now, substitute x=0: f(−1)+f(0)f(y)=−1, so f(0)f(y) takes the same value for all y∈R. Since f is not constant, this gives f(0)=0. We can then immediately conclude that f(−1)=−1. Now, substitute x=y=1: f(0)+f(1)2=1, so f(1)=1 or f(1)=−1.
Substituting y=1+x1 with x=0 gives f(x+1−1)+f(x)f(1+x1)=2x+2−1, so
f(x)f(1+x1)=2x+1−f(x) for all x=0
Substituting y=x1 with x=0 gives f(1−1)+f(x)f(x1)=2−1, so
f(x)f(x1)=1 for all x=0
Substituting y=1,x=z+1 gives f(z+1−1)+f(z+1)f(1)=2z+2−1, so
f(z)+f(z+1)f(1)=2z+1 for all z
We now choose z=x1 with x=0 in this equation and multiply the whole by f(x):
f(x)f(x1)+f(x1+1)f(1)f(x)=x2f(x)+f(x) for all x=0
We rewrite the first and second terms using (2) and (1) respectively, so we get:
1+2xf(1)+f(1)−f(x)f(1)=x2f(x)+f(x) for all x=0
This can be rewritten as
f(x)⋅(x2+1+f(1))=1+2xf(1)+f(1) for all x=0
If the second factor on the left is not 0, we can divide by it, so:
f(x)=x2+1+f(1)1+2xf(1)+f(1) if x=0 and x2+1+f(1)=0.
We had two possible values for f(1). First, let f(1)=1. Then we have
f(x)=x2+22+2x=x if x=0 and x2+2=0
x2+2=0 only if x=−1, but we already knew that f(−1)=−1. Also, we already had f(0)=0. We see that f(x)=x for all x. We check this function: in the original functional equation, we get xy−1+xy=2xy−1 on the left, so it works.
On the other hand, let f(1)=−1. Then we have
f(x)=x2−2x=−x2 if x=0 and x2=0
Since x2=0 cannot occur, we conclude that f(x)=−x2 for all x=0. Also, f(0)=0 satisfies this formula. We check the function f(x)=−x2 for all x: in the original functional equation, we get −x2y2−1+2xy+x2y2=2xy−1 on the left, so it works.
We conclude that there are two solutions, namely f(x)=x for all x and f(x)=−x2 for all x.