Iranian Mathematical Olympiad

First, note that the function is clearly strictly decreasing. Assume the supremum of the function is finite. In this case, for any $\epsilon > 0$, there exists $\delta$ such that for any $x < \delta$, we have $l - f(x) < \frac{\epsilon}{2}$. Now, in the given inequality, let $x, y$ tend to zero. We get $f(f(f(l))) \le 0$, which is a contradiction. Next, set $y = 1$ and let $x$ tend to zero. The right side tends to infinity, so the expression inside the left side must tend to zero, which implies that the infimum of the function is zero.

We have so far obtained that the limit of the function as $x \to 0$ is infinity, and as $x \to \infty$ is zero. Now, fix $x$ and let $y$ tends to zero. We obtain that $f(x) - f(x + y)$ must tend to zero, because otherwise the left side would be bounded while the right side would be unbounded. Therefore, the function is continuous. Now, set $y = \frac{k}{x}$ and let $x$ tend to zero. We obtain
$$f^3(k) = \lim_{x \to 0} f(x) f\left(\frac{k}{x}\right)$$
This implies that $f^3$ is multiplicative, that is
$$\frac{f^3(k)}{f^3(k')} = \lim_{x \to 0} \frac{f\left(\frac{k}{x}\right)}{f\left(\frac{k'}{x}\right)} = g\left(\frac{k}{k'}\right)$$
which implies
$$f^3(kk')f^3(1) = f^3(k)f^3(k')$$
Combined with the monotonicity and the limits of the function at zero and infinity, this implies $f^3(x) = \frac{c}{x^r}$, for some real $r$.

Now, choose $c$ such that $f(c) < 1$ and set $x = c$. We will have
$$f(f(f(cy)) + c^2) < f(y) \implies f(f(cy)) + c^2 > f(y)$$
which implies $f(f(y)) < Ay + B$. On the other hand, if $f(c) > 1$ and choose $y$ sufficiently large, we will have
$$f(y) < f(f(cy)) + c^2 \implies y < f(cy) + c^2$$
which implies $f(f(y)) > A'y + B'$. Therefore, $f^6(y) = x^{r^2}$ falls between two linear functions, which implies $f^3(y) = \frac{c}{x}$. Now, let $g(x) = f^2(x)$ be a strictly increasing function such that $g^3(x) = x$. Thus, we have $f^2(x) = x$, which combined with $f^3(x) = \frac{c}{x}$ implies $f(x) = \frac{c}{x}$. It is easy to verify that $f(x) = 1/x$ is the only function that satisfies the statement of the problem. ■

Solution 2:
As in the first solution, we need the fact that $f$ is strictly decreasing as well as $\lim_{x \to 0^+} f(x) = +\infty$ and $\lim_{x \to +\infty} f(x) = 0$. Taking into account the monotonicity, the function would be differentiable almost everywhere, thus there exists a point $z > 0$ such that $f'(z)$ exists. Letting $x = z$ and $y \to 0^+$ yielding we get
$$0 < f(z^2/2) < \lim_{x \to z^+} f(x^2) = \lim_{y \to 0^+} yf(y) \cdot \frac{f(z) - f(z+y)}{y} = -f'(z) \lim_{y \to 0^+} yf(y),$$
which implies that $\lim_{y \to 0^+} yf(y) = \ell_1$ exists and is positive. Let's rewrite the hypothesis as
$$(f(f(xy)) + x^2)f(f(f(xy)) + x^2) = \left(\frac{f(f(xy))}{xy} + \frac{x}{y}\right) yf(y)(xf(x) - xf(x+y)).$$
Since $\lim_{x \to 0^+} f(f(xy)) + x^2 = 0$, letting $x \to 0^+$ yielding
$$\ell_1 = \left( \lim_{x \to 0^+} \frac{f(f(xy))}{xy} \right) yf(y)\ell_1.$$
Since the inside limit is independent of $y$, we obtain at $yf(y) = \frac{1}{\ell_2}$, where $\ell_2 = \lim_{x \to 0^+} \frac{f(f(x))}{x}$ (which obviously must exist, precisely because of the above equation). Plugging $f(y) = \frac{1}{y\ell_2}$ yields $f(y) = \frac{1}{y}$.