Suppose $a_1 < a_2 < a_3 < a_4 < a_5$ are taken from $1, 2, \ldots, 20$. If $a_1, a_2, a_3, a_4, a_5$ are not adjacent to each other, then we have
$$1 \leq a_1 < a_2 - 1 < a_3 - 2 < a_4 - 3 < a_5 - 4 \leq 16,$$
from which we know that the number of ways to select five numbers not adjacent to each other from $1, 2, \ldots, 20$ is the same as selecting five different numbers from $1, 2, \ldots, 16$, i.e., $C_{16}^5$. Therefore, the required probability is
$$\frac{C_{20}^5 - C_{16}^5}{C_{20}^5} = 1 - \frac{C_{16}^5}{C_{20}^5} = \frac{232}{323}.$$
The answer is $\frac{232}{323}$.