HMMT November, 2021

Solution:

Assume without loss of generality that the side length of $ABCD$ is $1$ so that the area of the square is also $1$. This also means that $OM=ON=\frac{1}{2}$. As $A'B'MC'D'N$ is equiangular, it can be seen that $\angle A'NO=60^{\circ}$, and also by symmetry, that $A'B' \parallel AB$, so $\angle OA'B'=45^{\circ}$ and $\angle OA'N=75^{\circ}$. Therefore, $A'NO$ is a $45$-$60$-$75$ triangle, which has sides in ratio $2:1+\sqrt{3}:\sqrt{6}$, so we may compute that $A'O=\frac{\sqrt{6}}{1+\sqrt{3}} \cdot \frac{1}{2}=\frac{3\sqrt{2}-\sqrt{6}}{4}$.

Further, the area of $A'NO$ can be found by taking the altitude to $NO$, which has length $\frac{1}{2} \cdot \frac{\sqrt{3}}{1+\sqrt{3}}=\frac{3-\sqrt{3}}{4}$, so the area is $\frac{1}{2} \cdot \frac{1}{2} \cdot \frac{3-\sqrt{3}}{4}=\frac{3-\sqrt{3}}{16}$.

The area of $OA'B'$ is $\frac{1}{2}\left(\frac{3\sqrt{2}-\sqrt{6}}{4}\right)^2=\frac{6-3\sqrt{3}}{8}$.

Combining everything together, we can find that $[A'B'MC'D'N]=4[A'NO]+2[OA'B']=\frac{3-\sqrt{3}}{4}+\frac{6-3\sqrt{3}}{4}=\frac{9-4\sqrt{3}}{4}$.

Therefore, our answer is $9000-400+30+4=8634$.