Call 1, 2, 3 small numbers, 4, 5, 6 medium numbers, and 7, 8, 9 large numbers. There are $9!$ ways to fill in the first $3 \times 3$ block of the grid to contain each of the numbers 1 through 9. Without loss of generality suppose that the first row is 123, the second row is 456, and the third row is 789. Now consider filling in the second $3 \times 3$ block. The small numbers must appear in the second or third rows. If all three of them appear in the second row, then the medium numbers must appear in the third row, and the large numbers must appear in the first row. If two of them appear in the second row, then there are 3 ways to choose their positions, 3 ways to choose the position of the remaining small number in the third row, and 3 ways to choose the positions of the two remaining large numbers in the first row, for a total of 27 choices. Similarly, there is 1 possibility if none of the small numbers appears in the second row, and there are 27 if one of them appears in the second row, for a grand total of $1 + 27 + 1 + 27 = 56$ choices. In each case there are $3! = 6$ ways to assign the small, medium, or large numbers within their categories, so there are $56 \cdot 6^3$ ways to fill in the second $3 \times 3$ block. The placement of small, medium, and large numbers in the third $3 \times 3$ block are then determined except for a permutation of the three numbers in each row.

Altogether, there are therefore $9! \cdot 56 \cdot 6^3 \cdot 6^3$ possibilities. The prime factorization of this number is
$$(2^7 \cdot 3^4 \cdot 5 \cdot 7) \cdot (2^3 \cdot 7) \cdot (2^6 \cdot 3^6) = 2^{16} \cdot 3^{10} \cdot 5^1 \cdot 7^2.$$
The requested sum is $2 \cdot 16 + 3 \cdot 10 + 5 \cdot 1 + 7 \cdot 2 = 81$.