Serbian Mathematical Olympiad

Solution:

Substituting $y=0$ gives $f(x f(0))=f(0)$. If $f(0) \neq 0$, the expression $x f(0)$ takes all real values, so $f$ is a constant function, but that does not satisfy the conditions. Therefore, $f(0)=0$.

Setting $y=x$ we obtain $f(0)=f\left(x^{2}\right)-x^{2}$, i.e., $f\left(x^{2}\right)=x^{2}$. Thus, $f(x)=x$ for all $x \geqslant 0$.

Now let us substitute arbitrary $x, y<0$. Since then $f(x y)=x y$, we have $f(x f(y)-y f(x))=0$, which can hold only for $x f(y)-y f(x) \leqslant 0$. Analogously, $y f(x)-x f(y) \leqslant 0$ as well, so $y f(x)=x f(y)$, i.e., $f(x) / x=f(y) / y$. It follows that $f(x)=c x$ for all $x<0$, where $c$ is some constant.

Now for $x<0<y$ we obtain $f((1-c) x y)=f(x y)-x y=(c-1) x y$, i.e., $f(z)=-z$ for $z=(1-c) x y$. If $c=1$, then $f(x) \equiv x$, which is obviously a solution. On the other hand, for $c \neq 1$ we have $z \neq 0$ and hence $f(z) \in\{c z, z\}$, so it must then be that $c=-1$, which gives the function $f(x)=|x|$ for all $x$. This function is also a solution: we have already checked all cases except $x>0>y$, and for $x>0>y$ we have $-2 x y=f(-2 x y)=f(x f(y)-y f(x))=f(x y)-x y=-2 x y$.

Therefore, the solutions are the functions $f(x)=x$ and $f(x)=|x|$.