Suppose that f:R→R satisfies the relation in the problem, i.e.
f(f(x−y))=f(x)f(y)−f(x)+f(y)−xy,(1)
for all x,y∈R.
Put f(0)=a
By substituting x=y=0 into (1) we get
f(a)=a2(2)
By substituting x=y=0 into (1), from (2) we get
(f(x))2=x2+a2,∀x∈R(3)
Therefore (f(x))2=(f(−x))2∀x∈R, i.e.
(f(x)+f(−x))(f(x)−f(−x))=0,∀x∈R(4)
Suppose that there exist x0=0 such that f(x0)=f(−x0).
By substituting y=0 into (1), we get
f(f(x))=af(x)−f(x)+a,∀x∈R(5)
and by substituting x=0,y=−x into (1), we get
f(f(x))=af(−x)+f(−x)−a,∀x∈R.(6)
(5) and (6) give
a(f(x)−f(−x)+f(x)+f(−x)=2a,∀x∈R.(7)
By substituting x=x0 into (7), we get
f(x0)=a(∗)
On the other hand, from (3), we deduce that if f(x1)=f(x2) then x12=x22. Therefore, (*) implies that x0=0 which contradicts the supposition that x0=0. This contradiction proves that f(x)=f(−x), ∀x∈R. Therefore (4) shows that
f(x)=f(−x)∀x=0(8)
By substituting (8) into (7), we get
a(f(x)−1)=0∀x=0,
and we deduce from it that a=0, since if a=0 then f(x)=1,∀x=0 which contradicts (8).
So, from (3), we have
(f(x))2=x2∀x∈R(9)
Now suppose that there exists x0=0 such that f(x0)=x0. Then (5) implies that
x0=f(x0)=−f(f(x0))=−f(x0)=−x0.
This contradiction proves that f(x)=x,∀x=0.
Hence (9) shows that f(x)=−x,∀x∈R.
By direct verification, it is easily seen that this function satisfies the condition of the problem.
So the function f(x)=−x,∀x∈R, is the unique function satisfying the conditions of the problem.