, 2015

The constant function $f(x) = 0$ is a solution.
Let $f$ be a solution that is not identically $0$. We shall show that $f(x) = x$ for all $x$. Letting $x = 0$ in the given equation, we get
$$f(0)[f(yf(0) - 1) + 1] = 0.$$
Suppose $f(0) \neq 0$. Let $x = yf(0) - 1$. As $y$ ranges over all real numbers, so does $x$. Thus we get $f(x) = -1$ for all $x$. But this does not satisfy the given equation. So $f(0) = 0$.
Now suppose that $f(a) = 0$ for some $a \neq 0$. Then the original equation becomes $0 = a^2f(y)$ for all $y$, implying that $f(y) = 0$ for all $y$. This contradicts our assumption that $f$ is not identically $0$. Thus $f(x) = 0$ iff $x = 0$.
Letting $x = y = 1$ in the given equation, we have $f(f(1) - 1) = 0$. Thus $f(1) = 1$. When $x = 1$, the original equation becomes

$$f(x)f(f(x)) = x^2 \qquad (2)$$
Now replace $x$ by $x-1$ in (2) and apply (1) 3 times, and finally apply (2)
$$\begin{aligned} (x-1)^2 &= f(x-1)f(f(x-1)) = (f(x)-1)[f(f(x)-1)] \\ x^2 - 2x + 1 &= (f(x)-1)[f(f(x))-1] = f(x)f(f(x)) - f(x) - f(f(x)) + 1 \\ &= x^2 - f(x) - f(f(x)) + 1 \end{aligned}$$
Therefore
$$f(x) + f(f(x)) = 2x. \qquad (3)$$
Eliminating $f(f(x))$ from (2) and (3) gives
$$[x - f(x)]^2 = 0,$$
so that $f(x) = x$, as claimed. It is clear that this is also a solution.