Polynomial $P(x)$ has three zeros, let us denote them by $x_1$, $x_2$ and $x_3$.
According to Viète's formulas we have:
$$\begin{aligned} x_1 + x_2 + x_3 &= -a \\ x_1 x_2 + x_2 x_3 + x_3 x_1 &= b \\ x_1 x_2 x_3 &= 8. \end{aligned}$$
It follows $x_1^2 + x_2^2 + x_3^2 = a^2 - 2b$, and from the A-G inequality we get:
$$a^2 - 2b = x_1^2 + x_2^2 + x_3^2 \ge 3 \sqrt[3]{x_1^2 x_2^2 x_3^2} = 3 \sqrt[3]{64} = 12,$$
which finishes the proof.