Let f be any function satisfying
f(xy−1)+f(x)f(y)=2xy−1(1)
for all x,y∈R. Plug in x=0 in (1), we get f(−1)+f(x)f(0)=−1 for all x∈R.
If f(0)=0, then f is a constant function which does not satisfy the equation (1) for all x,y∈R and hence we get a contradiction. So f(0)=0. Plug x=y=1 in (1) to get f(1)2=1. Thus f(1)=1 or f(1)=−1.
Case 1: f(1)=1. Substitute x=xy and y=1 in (1) to obtain f(xy−1)+f(xy)=2xy−1 for all x,y∈R. Thus, by (1), we have
f(xy)=f(x)f(y)(2)
for all x,y∈R. Next, we substitute y=1 and x=x+1,y=1 in (1) to get
f(x−1)=2x−1−f(x)andf(x+1)=2x+1−f(x)(3)
for all x∈R. Now letting y=x in (1) and using (2) and (3), we then have
2x2−1=f(x2−1)+f(x)2=f(x−1)f(x+1)+f(x)2=(2x−1−f(x))(2x+1−f(x))+f(x)2=2f(x)2−4xf(x)+4x2−1
and hence 2(f(x)−x)2=0 which implies that f(x)=x for all x∈R.
Case 2: f(1)=−1. By the same substitutions as in case 1, we get f(xy)=−f(x)f(y), f(x−1)=2x−1+f(x) and f(x+1)=−(2x+1)+f(x) for all x,y∈R.
Now plugging y=x in (1) and using the previous three equations, we get
2x2−1=f(x2−1)+f(x)2=−f(x−1)f(x+1)+f(x)2=−(2x−1+f(x))(−(2x+1)+f(x))+f(x)2=2f(x)+4x2−1
and thus f(x)=−x2 for all x∈R.
We can easily check that both functions obtained from those two cases satisfy (1) for all x,y∈R. Hence, all functions satisfying (1) for all x,y∈R are f(x)=x for all x∈R or f(x)=−x2 for all x∈R. □