Croatian Mathematical Competitions

From the condition of the problem we conclude that the system of equations
$$y = ax^2 + b$$
$$y = x + \frac{1}{x}$$
has three solutions, i.e. that the cubic equation
$$ax^3 - x^2 + bx - 1 = 0$$
has three real solutions. Let us denote these solutions by $x_1, x_2, x_3$. Viète's formulas give
$$x_1 + x_2 + x_3 = \frac{1}{a},$$
$$x_1x_2 + x_2x_3 + x_3x_1 = \frac{b}{a},$$
$$x_1x_2x_3 = \frac{1}{a}.$$
The inequality $3ab < 1$ is equivalent to $\frac{1}{a^2} > 3 \cdot \frac{b}{a}$, which gives
$$(x_1 + x_2 + x_3)^2 > 3 (x_1x_2 + x_2x_3 + x_3x_1).$$
It is not hard to see that this is equivalent to
$$(x_1 - x_2)^2 + (x_2 - x_3)^2 + (x_3 - x_1)^2 > 0,$$
and that inequality holds since the points (and consequently their abscissas) are mutually distinct.