Put x=y=0 into equation (b) to get f(0)=0. Put y=−x into equation (b) to get f(−x)=−f(x) and deduce that f(x−y)=f(x)−f(y).
Let x∈R∖{0,1}. We have
f(x−11)=(x−1)2f(x−1)=(x−1)2f(x)−1
On the other hand,
f(x−11)=f(x−11+1)−1=f(x−1x)−1=(xx−1)2f(xx−1)−1=(xx−1)21−f(x1)−1=(x−1)2x2−f(x)−1=(x−1)22x−1−f(x)
We deduce from these two relations that for all x∈R∖{0,1}, we have f(x)−1=2x−1−f(x) which is equivalent to f(x)=x. Therefore, f(x)=x for all x∈R.
Conversely, this function satisfies all the conditions of the problem.