HMMT February

Let $(a, b, c):=(P(1), P(P(1)), P(P(P(1))))$, so $P(t)=a t^{2}+b t+c$ and we wish to maximize $P(c)$. Then we have that $$\begin{aligned} a & =P(1)=a+b+c \\ b & =P(a)=a^{3}+a b+c \\ c & =P(b)=a b^{2}+b^{2}+c \end{aligned}$$ The first equation implies $c=-b$. The third equation implies $b^{2}(a+1)=0$, so $a=-1$ or $b=0$. If $b=0$, then $(a, b, c)=(0,0,0)$. If $a=-1$, then $b=(-1)^{3}+(-1) b+(-b)$ or $b=-\frac{1}{3}$, so $c=\frac{1}{3}$ and $(a, b, c)=\left(-1,-\frac{1}{3}, \frac{1}{3}\right)$. The first tuple gives $P(c)=0$, while the second tuple gives $P(c)=-\frac{1}{3^{2}}-\frac{1}{3^{2}}+\frac{1}{3}=\frac{1}{9}$, which is the answer.