In each row or column, exactly one box is left empty. There are $4!=24$ ways to choose the empty spots. Once that has been done, there are 6 ways to choose which two rows have 2 purple balls each. Now, assume without loss of generality that boxes $(1,1)$, $(2,2),(3,3)$, and $(4,4)$ are the empty ones, and that rows 1 and 2 have two purple balls each. Let $A, B, C$, and $D$ denote the $2 \times 2$ squares in the top left, top right, bottom left, and bottom right corners, respectively (so $A$ is formed by the first two rows and first two columns, etc.). Let $a, b, c$, and $d$ denote the number of purple balls in $A, B, C$, and $D$, respectively. Then $0 \leq a, d \leq 2, a+b=4$, and $b+d \leq 4$, so $a \geq d$. Now suppose we are given the numbers $a$ and $d$, satisfying $0 \leq d \leq a \leq 2$. Fortunately, the numbers of ways to color the balls in $A, B, C$, and $D$ are independent of each other. For example, given $a=1$ and $d=0$, there are 2 ways to color $A$ and 1 way to color $D$ and, no matter how the coloring of $A$ is done, there are always 2 ways to color $B$ and 3 ways to color $C$. The numbers of ways to choose the colors of all the balls is as follows: $$\begin{array}{c|c|c|c} a \backslash d & 0 & 1 & 2 \\ \hline 0 & 1 \cdot(1 \cdot 2) \cdot 1=2 & 0 & 0 \\ \hline 1 & 2 \cdot(2 \cdot 3) \cdot 1=12 & 2 \cdot(1 \cdot 1) \cdot 2=4 & 0 \\ \hline 2 & 1 \cdot(2 \cdot 2) \cdot 1=4 & 1 \cdot(3 \cdot 2) \cdot 2=12 & 1 \cdot(2 \cdot 1) \cdot 1=2 \end{array}$$ In each square above, the four factors are the number of ways of arranging the balls in $A$, $B, C$, and $D$, respectively. Summing this over all pairs $(a, d)$ satisfying $0 \leq d \leq a \leq 2$ gives a total of 36. The answer is therefore $24 \cdot 6 \cdot 36=5184$.