, 2008

We show that $f(x) = 1/x$ is the only solution. We do this in several steps.

Step 1 We show that $f(x)$ is a non-increasing function. Suppose the contrary; say $0 < a < b$ implies that $f(a) < f(b)$, for some $a, b$. Then
$$w = \frac{b f(b) - a f(a)}{b - a}$$
is in $(0, \infty)$. It is easy to check that $w > f(b)$ using $f(b) > f(a)$. Thus $w > f(b) > f(a)$. Taking $x = a$ and $y = w - f(a)$, we get
$$f(w) = a f \left( 1 + a (w - f(a)) \right).$$
Similarly, $x = b$ and $y = w - f(b)$ gives
$$f(w) = b f \left( 1 + b (w - f(b)) \right).$$
Thus $a = b$ contradicting $a < b$. We conclude that $a < b$ implies that $f(b) \le f(a)$.

Step 2 Taking $x = y = 1$, we get $f(f(1) + 1) = f(2)$. Similarly, putting $x = 1, y = 2$, we obtain $f(f(1) + 2) = f(3)$; $x = 2, y = 1$ implies $f(f(2) + 1) = 2 f(3)$. Thus
$$\begin{aligned} 2 f(3) &= f(f(2) + 1) = f(f(f(1) + 1) + 1) \\ &= (f(1) + 1) f(1 + (f(1) + 1)) = (f(1) + 1) f(f(1) + 2) = (f(1) + 1) f(3). \end{aligned}$$
This gives $f(1) + 1 = 2$ and hence $f(1) = 1$.

Step 3 Suppose $x > 1$, and put $y = 1 - (1/x)$. Then
$$f\left(f(x) - \frac{1}{x} + 1\right) = x f(1 + x - 1) = x f(x).$$
If $f(x) > (1/x)$, we see that $f(x) - (1/x) + 1 > 1$ and the monotonicity of $f(x)$ gives
$$f\left(f(x) - \frac{1}{x} + 1\right) \le f(1) = 1.$$
Thus $x f(x) \le 1$, contradicting $f(x) > (1/x)$.
If $f(x) < (1/x)$, we see that $f(x) - (1/x) + 1 < 1$, and
$$f\left(f(x) - \frac{1}{x} + 1\right) \ge f(1) = 1,$$
and hence
$$1 \le f\left(f(x) - \frac{1}{x} + 1\right) = x f(x) < 1,$$
which again is impossible. We conclude that $f(x) = \frac{1}{x}$, for all $x > 1$.

Now, take any $x > 0$. Then $f(x) + 1 > 1$, so that
$$f(f(x) + 1) = \frac{1}{f(x) + 1}.$$
Putting $y = 1$ in the given relation, we get
$$f(f(x) + 1) = x f(1 + x) = \frac{x}{1 + x};$$
we have used $1 + x > 1$ in the last equality. Thus we obtain
$$\frac{1}{f(x) + 1} = \frac{x}{1 + x},$$
for all $x > 0$. This gives $f(x) = 1/x$ for all $x > 0$.