Denote by $X$ the set of all matrices with entries in the set $\{1,2,3,..., p\}$ and let $A_i$, respectively $B_j$, the set of matrices in which the sum of elements in the row $i$, respectively on the column $j$, be divisible by $p$. Then we have to obtain the cardinality $N$ of the set
$$X \setminus \left( \left( \bigcup_{i=1}^{m} A_i \right) \cup \left( \bigcup_{j=1}^{n} B_j \right) \right).$$
For this we apply the inclusion-exclusion principle. Then
$$N = p^{mn} + \sum_{i=0}^{m} \sum_{j=0}^{n} (-1)^{i+j} \sum_{\substack{k_1 < k_2 < \dots < k_i \\ l_1 < l_2 < \dots < l_j}} |A_{k_1} \cap A_{k_2} \cap \dots \cap A_{k_i} \cap B_{l_1} \cap B_{l_2} \cap \dots \cap B_{l_j}| \quad (i)$$
the sum being taken for all $i, j$ with $i + j \neq 0$.
But we have
$$|A_{k_1} \cap A_{k_2} \cap \dots \cap A_{k_i} \cap B_{l_1} \cap B_{l_2} \cap \dots \cap B_{l_j}| = |A_1 \cap A_2 \cap \dots \cap A_i \cap B_1 \cap B_2 \cap \dots \cap B_j| \quad (ii)$$
and moreover
$$p^{mn} = (-1)^{0+0} \binom{m}{0} \binom{n}{0} p^{mn-0-0}.$$
For all $i, j$ participating in the summation we have
$$|A_1 \cap A_2 \cap \dots \cap A_i \cap B_1 \cap B_2 \cap \dots \cap B_j| = \begin{cases} p^{mn-i-j}, & \text{if } i \neq m \text{ or } j \neq n \\ p^{mn-m-n-1}, & \text{if } i = m \text{ and } j = n \end{cases}$$
So,
$$\begin{align*} N &= \sum_{i=0}^{m} \sum_{j=0}^{m} \sum_{i+j \neq m+n} (-1)^{i+j} \binom{m}{i} \binom{n}{j} p^{mn-i-j} + (-1)^{m+n} \binom{m}{m} \binom{n}{n} p^{mn-m-n+1} = \\ &= \sum_{i=0}^{m} \sum_{j=0}^{m} (-1)^{i+j} \binom{m}{i} \binom{n}{j} p^{mn-i-j} - (-1)^{m+n} \binom{m}{m} \binom{n}{n} p^{mn-m-n} + \\ & \qquad + (-1)^{m+n} \binom{m}{m} \binom{n}{n} p^{mn-m-n+1} = \end{align*}$$
---

$$\begin{align*} &= \sum_{s=0}^{m+n} (-1)^s p^{mn-s} \sum_{i+j=s} \binom{m}{i} \binom{n}{j} + (-1)^{m+n} \binom{m}{m} \binom{n}{n} p^{mn-m-n} (p-1) = \\ &= p^{mn} \sum_{s=0}^{m+n} \left(\frac{-1}{p}\right)^s \binom{m+n}{s} + (-1)^{m+n} p^{mn-m-n} (p-1) = \\ &= p^{mn} \left( \left(-\frac{1}{p}\right)^{m+n} + (-1)^{m+n} \binom{m}{m} \binom{n}{n} p^{mn-m-n} (p-1) \right) = \\ &= p^{mn-m-n} \left( (p-1)^{m+n} + (-1)^{m+n} (p-1) \right) \end{align*}$$
$$$$