HMMT November, 2013

Solution:
Answer: $\frac{96 \sqrt{3}-154}{\sqrt{3}}$ OR $\frac{288-154 \sqrt{3}}{3}$ OR $96-\frac{154}{\sqrt{3}}$ OR $96-\frac{154 \sqrt{3}}{3}$

Let the rectangle be $ABCD$ with $AB=8$ and $BC=6$. Let the four equilateral triangles be $ABP_1$, $BCP_2$, $CDP_3$, and $DAP_4$ (for convenience, call them the $P_1$, $P_2$, $P_3$, $P_4$ triangles). Let $W=AP_1 \cap DP_3$, $X=AP_1 \cap DP_4$, and $Y=DP_4 \cap CP_2$. Reflect $X, Y$ over the line $P_2P_4$ (the line halfway between $AB$ and $DC$) to points $X', Y'$.

First we analyze the basic configuration of the diagram. Since $AB=8<2 \cdot 6 \frac{\sqrt{3}}{2}$, the $P_2$, $P_4$ triangles intersect. Furthermore, $AP_1 \perp BP_2$, so if $T=BP_2 \cap AP_1$, then $BP_2=6<4\sqrt{3}=BT$. Therefore $P_2$ lies inside triangle $P_1BA$, and by symmetry, also triangle $P_3DC$.

It follows that the area we wish to compute is the union of two (congruent) concave hexagons, one of which is $WXY P_2 Y' X'$. (The other is its reflection over $YY'$, the mid-line of $AD$ and $BC$.) So we seek
$$2\left[WXYP_2Y'X'\right]=2\left(\left[WXP_4X'\right]-\left[P_2YP_4Y'\right]\right)$$
It's easy to see that $\left[WXP_4X'\right]=\frac{1}{3}\left[ADP_4\right]=\frac{1}{3} \frac{6^2 \sqrt{3}}{4}=3\sqrt{3}$, since $WXP_4X'$ and its reflections over lines $DWX'$, $AWX$ partition $\triangle ADP_4$.

It remains to consider $P_2YP_4Y'$, a rhombus with (perpendicular) diagonals $P_2P_4$ and $YY'$. If $O$ denotes the intersection of these two diagonals (also the center of $ABCD$), then $OP_2$ is $P_2B \frac{\sqrt{3}}{2}-\frac{1}{2}AB=3\sqrt{3}-4$, the difference between the lengths of the $P_2$-altitude in $\triangle CBP_2$ and the distance between the parallel lines $YY'$, $CB$. Easy angle chasing gives $OY=\frac{OP_2}{\sqrt{3}}$, so
$$\left[P_2YP_4Y'\right]=4 \cdot \frac{OP_2 \cdot OY}{2}=\frac{2}{\sqrt{3}} OP_2^2=\frac{2}{\sqrt{3}}(3\sqrt{3}-4)^2=\frac{86-48\sqrt{3}}{\sqrt{3}}$$
and our desired area is
$$2\left[WXP_4X'\right]-2\left[P_2YP_4Y'\right]=6\sqrt{3}-\frac{172-96\sqrt{3}}{\sqrt{3}}=\frac{96\sqrt{3}-154}{\sqrt{3}}$$
or $\frac{288-154\sqrt{3}}{3}$.