Estonian Mathematical Olympiad

Substituting $y = 1$ into the given equation, we obtain
$$f(0) = f(1) + 2(x^2 - f(x))(f(1) - 1) + 1$$
which is equivalent to
$$2f(x)(f(1) - 1) = f(1) - f(0) + 2x^2(f(1) - 1). \quad (3)$$

If $f(1) = 1$ then (3) implies $0 = 1 - f(0) + 1$, or equivalently, $f(0) = 2$.
Substituting now $y = 0$ into the original equation and applying $f(0) = 2$ gives us
$$f(-2) = 2 + 2(x^2 - f(x)) + 1,$$
or equivalently, $f(x) = x^2 + c$ where $c = \frac{3-f(-2)}{2}$. In the other case, $f(x)$ is expressed in the same form with $c = \frac{f(1)-f(0)+1}{2(f(1)-1)}$.
We show that the function $f(x) = x^2 + c$ satisfies the original equation if and only if $c = 1$. Substituting $f(x) = x^2 + c$ into the original equation and simplifying leads to
$$(c^2 - 1) y^4 + 2c(1-c)y^2 + (3c^2 - 2c - 1) = 0.$$
This equality must hold for every real number $y$. For that, all coefficients in the left hand side must be zeros. From the leading term, we get $c^2 - 1 = 0$, implying $c = 1$ or $c = -1$. From the quadratic term, we get $2c(1-c) = 0$, implying $c = 0$ or $c = 1$. Altogether, only $c = 1$ works. It makes the constant term also zero. Hence $f(x) = x^2 + 1$ is the only function that satisfies the given equation.

Substituting $y = 0$ into the original equation gives
$$f(-f(0)) = f(0) + 2(x^2 - f(x))(f(0) - 1) + 1,$$
or equivalently,
$$2f(x)(f(0)-1) = f(0) - f(-f(0)) + 1 + 2x^2(f(0)-1). \quad (4)$$
This implies that either $f(0) = 1$ or $f(x) = x^2 + \frac{f(0)-f(-f(0))+1}{2(f(0)-1)}$ for any real number $x$.

Consider the case $f(0) = 1$. Substituting $x = 0$ into the original equation leads to
$$f(y^2 - 1) = f(y^2) - 2(f(y) - 1) + 1,$$
which must hold for any real number $y$. Substituting $-y$ for $y$ gives
$$f(y^2 - 1) = f(y^2) - 2(f(-y) - 1) + 1,$$
which must also hold for any real number $y$. Hence $f(y) = f(-y)$ for any real number $y$, i.e., $f$ is an even function. Eliminating the two terms with $f(0) - 1$ in (4) gives $0 = 1 - f(-1) + 1$, or equivalently, $f(-1) = 2$. Substituting now $y = -1$ into the original equation gives
$$f(f(x) - f(-x)) = f(1) + 2(x^2 - f(x))(f(-1) - 1) + 1. \quad (5)$$
As $f$ is an even function and $f(1) = f(-1) = 2$, the equation (5) simplifies to $1 = 2 + 2(x^2 - f(x)) + 1$ which is equivalent to $f(x) = x^2 + 1$.

Hence $f(x) = x^2 + c$ for any real number $x$, where $c$ is some constant. We proceed like in Solution 1.