More generally, suppose $a^{2}+a b+b^{2}=z, b^{2}+b c+c^{2}=x$, $c^{2}+c a+a^{2}=y$ for some complex numbers $a, b, c, x, y, z$. We show that $$f(a, b, c, x, y, z)=\left(\frac{1}{2}(a b+b c+c a) \sin 120^{\circ}\right)^{2}-\left(\frac{1}{4}\right)^{2}\left[(x+y+z)^{2}-2\left(x^{2}+y^{2}+z^{2}\right)\right]$$ holds in general. Plugging in $x=-2, y=1, z=1+i$ will then yield the desired answer, $$\begin{aligned} (a b+b c+c a)^{2} & =\frac{16}{3} \frac{1}{16}\left[(x+y+z)^{2}-2\left(x^{2}+y^{2}+z^{2}\right)\right] \\ & =\frac{i^{2}-2\left(4+1+(1+i)^{2}\right)}{3}=\frac{-1-2(5+2 i)}{3}=\frac{-11-4 i}{3} \end{aligned}$$