Label the bottom left corner of the larger rectangle (without the square cut out) as $A$ and the top right as $D$. $w$ is the width of the rectangle and $\ell$ is the length. Now we have vertices $E, F, G, H$ as vertices of the irregular octagon created by cutting out the squares. Let $I, J$ be the two closest vertices formed by the squares.
The distance between the two closest vertices of the squares is thus $IJ=\left(4\sqrt{2}\right).$
Substituting, we get
$$(IJ)^2 = (w-2)^2 + (\ell-2)^2 = \left(4\sqrt{2}\right)^2 = 32 \implies w^2+\ell^2-4w-4\ell = 24.$$
Using the fact that the diagonal of the rectangle is $8,$ we get
$$w^2+\ell^2 = 64.$$
Subtracting the first equation from the second equation, we get $$4w+4\ell=40 \implies w+\ell = 10.$$
Squaring yields $$w^2 + 2w\ell + \ell^2 = 100.$$
Subtracting the second equation from this, we get $2w\ell = 36,$ and thus area of the original rectangle is $w\ell = \boxed{\textbf{(E) } 18}.$
~USAMO333
Edits and Diagram by ~KingRavi and ~MRENTHUSIASM
Minor edit by yanes04