Solution:
The prime factorization of $9!$ is $2^{7} \cdot 3^{4} \cdot 5 \cdot 7$. Every divisor of $9!$ has prime factorization $2^{a} \cdot 3^{b} \cdot 5^{c} \cdot 7^{d}$, where $0 \leq a \leq 7$, $0 \leq b \leq 4$, $0 \leq c \leq 1$, and $0 \leq d \leq 1$. If the divisor has units digit $1$, it cannot be divisible by $2$ or $5$, so $a = c = 0$.

Now take cases on the value of $d$:
- If $d = 0$, then the divisor is $3^{b}$ for some $0 \leq b \leq 4$. The possible divisors are $1$, $3$, $9$, $27$, and $81$, of which $1$ and $81$ work.
- If $d = 1$, then the divisor is $3^{b} \cdot 7$ for some $0 \leq b \leq 4$. The possible divisors are then $7$, $3 \cdot 7$, $9 \cdot 7$, $27 \cdot 7$, and $81 \cdot 7$. Of these, only $3 \cdot 7 = 21$ works.

The answer is $1 + 21 + 81 = \boxed{103}$.