HMMT February

For a point $P=(x, y)$, let $z(P)=x+y \omega$, where $\omega$ is a nontrivial third root of unity. Then $$\begin{aligned} z(f(P))=\left(x^{2}-y^{2}\right)+\left(2 x y-y^{2}\right) \omega=x^{2}+2 x y \omega+y^{2} & (-1-\omega) \\ & =x^{2}+2 x y \omega+y^{2} \omega^{2}=(x+y \omega)^{2}=z(P)^{2} \end{aligned}$$ Applying this recursively gives us $z\left(f^{n}(P)\right)=z\left(f^{n-1}(P)\right)^{2}=z\left(f^{n-2}(P)\right)^{4}=\cdots=z(P)^{2^{n}}$. Thus the condition $f^{n}(P) \rightarrow(0,0)$ is equivalent to $|z(P)|<1$. The region of such points is the preimage of the unit disk (area $\pi$) upon the "shear" sending $(0,1)$ to $\left(-\frac{1}{2}, \frac{\sqrt{3}}{2}\right)$. This shear multiplies areas by a factor of $\frac{\sqrt{3}}{2}$, so the original area was $\frac{2 \pi}{\sqrt{3}}=\pi \sqrt{\frac{4}{3}}$.