HMMT, 2014

Solution:
Answer: $\quad \frac{-11-4 i}{3}$ OR $-\frac{11+4 i}{3}$

More generally, suppose $a^{2}+a b+b^{2}=z$, $b^{2}+b c+c^{2}=x$, $c^{2}+c a+a^{2}=y$ for some complex numbers $a, b, c, x, y, z$.
We show that
$$f(a, b, c, x, y, z)=\left(\frac{1}{2}(a b+b c+c a) \sin 120^{\circ}\right)^{2}-\left(\frac{1}{4}\right)^{2}\left[(x+y+z)^{2}-2\left(x^{2}+y^{2}+z^{2}\right)\right]$$
holds in general. Plugging in $x=-2, y=1, z=1+i$ will then yield the desired answer,
$$\begin{aligned} (a b+b c+c a)^{2} & =\frac{16}{3} \frac{1}{16}\left[(x+y+z)^{2}-2\left(x^{2}+y^{2}+z^{2}\right)\right] \\ & =\frac{i^{2}-2\left(4+1+(1+i)^{2}\right)}{3}=\frac{-1-2(5+2 i)}{3}=\frac{-11-4 i}{3} \end{aligned}$$

Solution 1: Plug in $x=b^{2}+b c+c^{2}$, etc. to get a polynomial $g$ in $a, b, c$ (that agrees with $f$ for every valid choice of $a, b, c, x, y, z$). It suffices to show that $g(a, b, c)=0$ for all positive reals $a, b, c$, as then the polynomial $g$ will be identically $0$.
But this is easy: by the law of cosines, we get a geometrical configuration with a point $P$ inside a triangle $A B C$ with $P A=a, P B=b, P C=c, \angle P A B=\angle P B C=\angle P C A=120^{\circ}, x=B C^{2}$,
$y=C A^{2}, z=A B^{2}$. By Heron's formula, we have
$$\begin{aligned} \left(\frac{1}{2}(a b+b c+c a) \sin 120^{\circ}\right)^{2} & =[A B C]^{2} \\ & =\frac{(\sqrt{x}+\sqrt{y}+\sqrt{z}) \prod_{\mathrm{cyc}}(\sqrt{x}+\sqrt{y}-\sqrt{z})}{2^{4}} \\ & =\frac{1}{16}\left[(\sqrt{x}+\sqrt{y})^{2}-z\right]\left[(\sqrt{x}-\sqrt{y})^{2}-z\right] \\ & =\frac{1}{16}\left[(x-y)^{2}+z^{2}-2 z(x+y)\right] \\ & =\left(\frac{1}{4}\right)^{2}\left[(x+y+z)^{2}-2\left(x^{2}+y^{2}+z^{2}\right)\right] \end{aligned}$$
as desired.

Solution 2: Let $s=a+b+c$. We have $x-y=b^{2}+b c-c a-a^{2}=(b-a) s$ and cyclic, so $x+a s=y+b s=z+c s$ (they all equal $\sum a^{2}+\sum b c$ ). Now add all equations to get
$$x+y+z=2 \sum a^{2}+\sum b c=s^{2}+\frac{1}{2} \sum(b-c)^{2}$$
multiplying both sides by $4 s^{2}$ yields $4 s^{2}(x+y+z)=4 s^{4}+2 \sum(z-y)^{2}$, so $\left[2 s^{2}-(x+y+z)\right]^{2}=$ $(x+y+z)^{2}-2 \sum(z-y)^{2}=6 \sum y z-3 \sum x^{2}$. But $2 s^{2}-(x+y+z)=s^{2}-\frac{1}{2} \sum(b-c)^{2}=(a+b+c)^{2}-\frac{1}{2} \sum(b-c)^{2}=3(a b+b c+c a)$, so
$$9(a b+b c+c a)^{2}=6 \sum y z-3 \sum x^{2}=3\left[(x+y+z)^{2}-2\left(x^{2}+y^{2}+z^{2}\right)\right]$$
which easily rearranges to the desired.