Problem 1246

We will use the well-known inequality $2\alpha\beta \le \alpha^2 + \beta^2$, which is valid for all $\alpha, \beta \in \mathbb{R}$. The equality holds for $\alpha = \beta$. Thus we have

$$\begin{align*} x^2 y^2 + y^2 z^2 + z^2 x^2 + xyz &= \frac{1}{2}(2x^2 y^2 + 2y^2 z^2 + 2z^2 x^2 + 2xyz) \\ &= \frac{1}{2}(2xy \cdot xy + 2yz \cdot yz + 2zx \cdot zx + 2xyz) \\ &\le \frac{1}{2}[xy(x^2 + y^2) + yz(y^2 + z^2) + zx(z^2 + x^2) + 2xyz] \tag{1} \\ &= \frac{1}{2}[(xy + yz + zx)(x^2 + y^2 + z^2) - xyz^2 - yzx^2 - zxy^2 + 2xyz] \\ &= \frac{1}{2}[(xy + yz + zx)(x^2 + y^2 + z^2) - xyz(x + y + z - 2)] \\ &= \frac{1}{2}[(xy + yz + zx)(x^2 + y^2 + z^2)], \quad (\text{since } x + y + z = 2). \end{align*}$$
Till now we have shown that
$$x^2 y^2 + y^2 z^2 + z^2 x^2 + xyz \le \frac{1}{2} \left[ (xy + yz + zx)(x^2 + y^2 + z^2) \right], \quad (2)$$
and the equality holds, as we see from (1), when:
$x = y = z$ or $x = y$, $z = 0$ or $y = z$, $x = 0$ or $z = x$, $y = 0$.
Since $x + y + z = 2$, equality holds when:
$$(x, y, z) = \left(\frac{2}{3}, \frac{2}{3}, \frac{2}{3}\right) \quad \text{or}~(1, 1, 0) \quad \text{or}~(1, 0, 1) \quad \text{or}~(0, 1, 1). \qquad (3)$$
In the sequel we will use the known inequality $\alpha\beta \le \left(\frac{\alpha + \beta}{2}\right)^2$, $\alpha, \beta \in \mathbb{R}$,
putting $\alpha = 2xy + 2yz + 2zx$, $\beta = x^2 + y^2 + z^2$. Thus we have
$$\begin{align*} \frac{1}{2} \left[ (xy + yz + zx)(x^2 + y^2 + z^2) \right] &= \frac{1}{4} \left[ (2xy + 2yz + 2zx)(x^2 + y^2 + z^2) \right] \\ &\le \frac{1}{4} \left( \frac{2xy + 2yz + 2zx + x^2 + y^2 + z^2}{2} \right)^2 = \frac{1}{16} (x+y+z)^4 = 1. \tag{4} \end{align*}$$
From (2) and (4) we obtain the inequality
$x^2 y^2 + y^2 z^2 + z^2 x^2 + xyz \le 1.$
The equality holds when in inequality (4) we have:
$\alpha = \beta \Leftrightarrow 2xy + 2yz + 2zx = x^2 + y^2 + z^2,$
which in consideration with (3) gives:
$$(x, y, z) = (1, 1, 0) \quad \text{or}~(1, 0, 1) \quad \text{or}~(0, 1, 1).$$