First Solution. a) Use AM-GM and xyz=8 to get
x2+y2+z2+xy+xy+yz+yz+xz+xz⩾99x6y6z6=36
We have equality for x=y=z=2.
b) Using xyz=8, we can transform the given expression:
x2+y2+z2+mxy+nxz+pyz=x2+x8p+y2+y8n+z2+z8m
Since all numbers are positive reals, we can apply AM-GM inequality to get:
x2+x8p=x2+x4p+x4p⩾632p2
When we apply the same procedure for x,y,z and sum the inequalities, we get:
x2+y2+z2+mxy+nxz+pyz=x2+x8p+y2+y8n+z2+z8m⩾632(3m2+3n2+3p2)
In order to get equality, we must have equality in all above inequalities and that happens for
x=34py=34nz=34m
Desired minimum is therefore
632(3m2+3n2+3p2)
Second Solution. We only present solution for b ) part here, marking scheme for a ) part is the same as in first solution. We use weighted AM-GM:
\begin{aligned}
& x^{2}+y^{2}+z^{2}+m x y+n x z+p y z= \\
& \sqrt[3]{p^{2}} \frac{x^{2}}{\sqrt[3]{p^{2}}}+\sqrt[3]{n^{2}} \frac{y^{2}}{\sqrt[3]{n^{2}}}+\sqrt[3]{m^{2}} \frac{z^{2}}{\sqrt[3]{m^{2}}}+2 \sqrt[3]{m^{2}} \frac{m x y}{2 \sqrt[3]{m^{2}}}+2 \sqrt[3]{n^{2}} \frac{n x z}{2 \sqrt[3]{n^{2}}}+2 \sqrt[3]{p^{2}} \frac{p y z}{2 \sqrt[3]{p^{2}}} \geq \\
& 3\left(\sqrt[3]{m^{2}}+\sqrt[3]{n^{2}}+\sqrt[3]{p^{2}}\right) \cdot \sqrt[3\left(\sqrt[3]{m^{2}}+\sqrt[3]{n^{2}}+\sqrt[3]{p^{2}}\right)]{\left(\frac{x^{2}}{\sqrt[3]{p^{2}}}\right)^{\sqrt[3]{p^{2}}}\left(\frac{y^{2}}{\sqrt[3]{n^{2}}}\right)^{\sqrt[3]{n^{2}}}\left(\frac{z^{2}}{\sqrt[3]{m^{2}}}\right)^{\sqrt[3]{m^{2}}}} \\
& 3\left(\sqrt[3]{m^{2}}+\sqrt[3]{n^{2}}+\sqrt[3]{p^{2}}\right) \sqrt{\left(\frac{\sqrt[3]{m} x y}{2}\right)^{2 \sqrt[3]{m^{2}}}\left(\frac{\sqrt[3]{n} x z}{2}\right)^{2 \sqrt[3]{n^{2}}}\left(\frac{\sqrt[3]{p} y z}{2}\right)^{2 \sqrt[3]{p^{2}}}} \\
& =3\left(\sqrt[3]{m^{2}}+\sqrt[3]{n^{2}}+\sqrt[3]{p^{2}}\right) \cdot \sqrt[3\left(\sqrt[3]{m^{2}}+\sqrt[3]{n^{2}}+\sqrt[3]{p^{2}}\right)]{\left(\frac{x y z}{2}\right)^{2\left(\sqrt[3]{m^{2}}+\sqrt[3]{n^{2}}+\sqrt[3]{p^{2}}\right)}} \\
& =3\left(\sqrt[3]{m^{2}}+\sqrt[3]{n^{2}}+\sqrt[3]{p^{2}}\right) \cdot \sqrt[3]{\left(\frac{x y z}{2}\right)^{2}}=3\left(\sqrt[3]{m^{2}}+\sqrt[3]{n^{2}}+\sqrt[3]{p^{2}}\right) \cdot \sqrt[3]{4^{2}}=6 \sqrt[3]{2}\left(\sqrt[3]{m^{2}}+\sqrt[3]{n^{2}}+\sqrt[3]{p^{2}}\right)
\end{aligned}
We have shown that the minimum value the expression can take is 632(3m2+3n2+3p2). Equality can only be achieved when x=34p,y=34n,z=34m.