Indian National Mathematical Olympiad

Solution:
We have from the AM-GM inequality, that
$$x y \leq \left(\frac{x + y}{2}\right)^{2} = 1$$
Thus we obtain $0 < x y \leq 1$. We write
$$\begin{aligned} x^{3} y^{3} (x^{3} + y^{3}) &= (x y)^{3} (x + y) (x^{2} - x y + y^{2}) \\ &= 2 (x y)^{3} ((x + y)^{2} - 3 x y) \\ &= 2 (x y)^{3} (4 - 3 x y) \end{aligned}$$
Thus we need to prove that
$$(x y)^{3} (4 - 3 x y) \leq 1$$
Putting $z = x y$, this inequality reduces to
$$z^{3} (4 - 3 z) \leq 1$$
for $0 < z \leq 1$. We can prove this in different ways. We can put the inequality in the form
$$3 z^{4} - 4 z^{3} + 1 \geq 0$$
Here the expression in the LHS factors to $(z - 1)^{2} (3 z^{2} + 2 z + 1)$ and $(3 z^{2} + 2 z + 1)$ is positive since its discriminant $D = -8 < 0$. Or applying the AM-GM inequality to the positive reals $4 - 3 z, z, z, z$, we obtain
$$z^{3} (4 - 3 z) \leq \left(\frac{4 - 3 z + 3 z}{4}\right)^{4} \leq 1$$