Belarus — Selection and Training Session

(Solution by A. Asanau, Y. Dubovik, Y. Laurenau, V. Vityaz.) Suppose that such $a$, $b$, $f$ exist. Obviously $b \neq 0$. Further, since $f$ is surjective, there exists a $\lambda$ such that $f(\lambda) = 0$. Set $x = \lambda$ in
$$f(f(x)) = b x f(x) + a, \tag{1}$$
then $f(0) = f(f(\lambda)) = b \lambda f(\lambda) + a = b \lambda \cdot 0 + a = a$. Further,
$$f(a) = f(f(0)) = b \cdot 0 \cdot f(0) + a = a.$$
Now $a = f(a) = f(f(a)) = b a f(a) + a = b a^2 + a$ whence $a = 0$. Thus (1) becomes
$$f(f(x)) = b x f(x). \tag{2}$$
Further, there exists a $\sigma \neq 0$ such that $f(\sigma) = -1/b$. Set $x = \sigma$ in (2), then
$$f(-1/b) = f(f(\sigma)) = b \sigma f(\sigma) = b \sigma(-1/b) = -\sigma. \quad (3)$$
Hence
$$f(-\sigma) = f(f(-1/b)) = b(-1/b)f(-1/b) = \sigma.$$
Therefore,
$$\frac{1}{b} = f(\sigma) = f(f(-\sigma)) = b(-\sigma)f(-\sigma) = -b \sigma^2$$
and so $\sigma = \pm 1/b$.
If $\sigma = -1/b$, then $\sigma = -1/b = f(\sigma) = f(-1/b)$, contrary to (3).
If $\sigma = 1/b$, then in view of (3) we have
$$f(-\sigma) = -\sigma \implies -\sigma = f(-\sigma) = f(f(-\sigma)) = b(-\sigma)f(-\sigma) = b \sigma^2 \implies \\ \qquad \sigma = -1/b,$$
a contradiction.

Therefore, there are no $a$, $b$, $f$ satisfying the problem condition.