China Mathematical Competition

(a) When $a < -2$, then $|f^1(0)| = |a| > 2$, therefore $a \notin M$.

(b) When $-2 \le a < 0$, we have $|f^1(0)| = |a| \le 2$. Assume that $|f^{k-1}(0)| \le |a| \le 2$ for $k \ge 2$. Since $a^2 \le -2a$ for $-2 \le a < 0$, we get that
$$-2 \le -|a| = a \le f^k(0) = (f^{k-1}(0))^2 + a \le a^2 + a \le |a| \le 2.$$
By the principle of mathematical induction, we conclude that $|f^n(0)| \le a \le 2$ ($\forall n \ge 1$).

(c) When $0 \le a \le \frac{1}{4}$, we have $|f^1(0)| = |a| \le \frac{1}{2}$. Assume that $|f^{k-1}(0)| \le \frac{1}{2}$ for $k \ge 2$. We get
$$|f^k(0)| \le |f^{k-1}(0)|^2 + a \le \left(\frac{1}{2}\right)^2 + \frac{1}{4} = \frac{1}{2}.$$
By the principle of mathematical induction, we conclude that $|f^n(0)| \le \frac{1}{2}$ ($\forall n \ge 1$).

From (b) and (c), we obtain $[-2, \frac{1}{4}] \subseteq M$.

(d) When $a > \frac{1}{4}$, define $a_n = f^n(0)$. We have
$$a_{n+1} = f^{n+1}(0) = f(f^n(0)) = f(a_n) = a_n^2 + a,$$
then $a_n > a > \frac{1}{4}$ for any $n \ge 1$. Since
$$a_{n+1} - a_n = a_n^2 - a_n + a = \left(a_n - \frac{1}{2}\right)^2 + a - \frac{1}{4} \ge a - \frac{1}{4},$$
we get
$$\begin{aligned} a_{n+1} - a &= a_{n+1} - a_1 \\ &= (a_{n+1} - a_n) + \dots + (a_2 - a_1) \\ &\ge n \left(a - \frac{1}{4}\right). \end{aligned}$$
Therefore, when $n > \frac{2-a}{a-\frac{1}{4}}$, we have
$$a_{n+1} \ge n\left(a - \frac{1}{4}\right) + a > 2 - a + a = 2.$$
And that means $a \notin M$.

From (a)—(d), we proved that $M = [-2, \frac{1}{4}]$.