Vietnam Mathematical Olympiad

Suppose that $f: \mathbb{R} \to \mathbb{R}$ satisfies the relation in the problem, i.e.
$$f(f(x - y)) = f(x)f(y) - f(x) + f(y) - xy, \quad (1)$$
for all $x, y \in \mathbb{R}$.
Put $f(0) = a$
By substituting $x = y = 0$ into (1) we get
$$f(a) = a^2 \quad (2)$$
By substituting $x = y = 0$ into (1), from (2) we get
$$(f(x))^2 = x^2 + a^2, \quad \forall x \in \mathbb{R} \quad (3)$$
Therefore $(f(x))^2 = (f(-x))^2 \quad \forall x \in \mathbb{R}$, i.e.
$$(f(x) + f(-x))(f(x) - f(-x)) = 0, \quad \forall x \in \mathbb{R} \quad (4)$$
Suppose that there exist $x_0 \neq 0$ such that $f(x_0) = f(-x_0)$.
By substituting $y = 0$ into (1), we get
$$f(f(x)) = af(x) - f(x) + a, \forall x \in \mathbb{R} \quad (5)$$
and by substituting $x = 0, y = -x$ into (1), we get
$$f(f(x)) = af(-x) + f(-x) - a, \forall x \in \mathbb{R}. \quad (6)$$
(5) and (6) give
$$a(f(x) - f(-x) + f(x) + f(-x) = 2a, \forall x \in \mathbb{R}. \quad (7)$$
By substituting $x = x_0$ into (7), we get
$$f(x_0) = a \quad (*)$$
On the other hand, from (3), we deduce that if $f(x_1) = f(x_2)$ then $x_1^2 = x_2^2$. Therefore, (*) implies that $x_0 = 0$ which contradicts the supposition that $x_0 \neq 0$. This contradiction proves that $f(x) \neq f(-x)$, $\forall x \in \mathbb{R}$. Therefore (4) shows that
$$f(x) = f(-x) \quad \forall x \neq 0 \quad (8)$$
By substituting (8) into (7), we get
$$a(f(x) - 1) = 0 \quad \forall x \neq 0,$$
and we deduce from it that $a = 0$, since if $a \neq 0$ then $f(x) = 1, \forall x \neq 0$ which contradicts (8).
So, from (3), we have
$$(f(x))^2 = x^2 \quad \forall x \in \mathbb{R} \quad (9)$$
Now suppose that there exists $x_0 \neq 0$ such that $f(x_0) = x_0$. Then (5) implies that
$$x_0 = f(x_0) = -f(f(x_0)) = -f(x_0) = -x_0.$$
This contradiction proves that $f(x) \neq x, \forall x \neq 0$.
Hence (9) shows that $f(x) = -x, \forall x \in \mathbb{R}$.
By direct verification, it is easily seen that this function satisfies the condition of the problem.

So the function $f(x) = -x, \forall x \in \mathbb{R}$, is the unique function satisfying the conditions of the problem.