1. Understanding the Set:
The set given is $\{10a + b : 1 \leq a \leq 5, 1 \leq b \leq 5\}$. This set consists of numbers formed by $10a + b$ where $a$ and $b$ are integers from 1 to 5. This results in a 5x5 grid of numbers:
$$\begin{array}{ccccc} 11 & 12 & 13 & 14 & 15 \\ 21 & 22 & 23 & 24 & 25 \\ 31 & 32 & 33 & 34 & 35 \\ 41 & 42 & 43 & 44 & 45 \\ 51 & 52 & 53 & 54 & 55 \\ \end{array}$$

2. Choosing 5 Numbers:
We need to choose 5 distinct numbers such that no two numbers have the same units digit and no two numbers have the same tens digit. This means we are essentially choosing one number from each row and each column of the 5x5 grid.

3. Counting the Number of Ways:
The problem reduces to counting the number of ways to select one number from each row and each column of a 5x5 grid. This is equivalent to finding the number of permutations of 5 elements, which is $5!$.

4. **Calculating $5!$**:
$$5! = 5 \times 4 \times 3 \times 2 \times 1 = 120$$

5. Finding the Remainder:
We need to find the remainder when $120$ is divided by $73$:
$$120 \div 73 = 1 \quad \text{remainder} \quad 47$$
Therefore, the remainder is $47$.

The final answer is $\boxed{47}$