, 2010

Since we are given that $0 \le a, b \le 1$ holds, it follows from the AM-GM inequality, that
$$\begin{aligned} & \sqrt{a^3 b^3} + \sqrt{(1-a^2)(1-ab)(1-b^2)} \\ & \le \sqrt[3]{a^3 b^3} + \sqrt[3]{(1-a^2)(1-ab)(1-b^2)} \\ & = \sqrt[3]{a^2 \cdot ab \cdot b^2} + \sqrt[3]{(1-a^2)(1-ab)(1-b^2)} \\ & \le \frac{a^2 + ab + b^2}{3} + \frac{(1-a^2) + (1-ab) + (1-b^2)}{3} \\ & = 1, \end{aligned}$$
as claimed. We note that equality holds iff $a^2 = ab = b^2$ and either $a^3b^3 = 0$ or $a^3b^3 = 1$ holds (and therefore also the same for $(1-a^2)(1-ab)(1-b^2)$). Equality therefore holds iff $a = b = 0$ or $a = b = 1$.