The minimum possible score is $10$.
Suppose we have a partition $S_1, S_2, \dots, S_n$ with score $x$.

Claim 1. If there are at least two subsets, say $S_1, S_2$, both with averages at least $x$, then merging these two subsets into one subset will not increase the score.
Proof. Let $a'$ be the new average of the merged set. Note that $a' \le \max\{a_1, a_2\}$. Together with $x \le a_1, a_2$, we obtain $a' + x \le a_1 + a_2$. It follows that
$$\frac{a' + a_3 + a_4 + \dots + a_n}{n-1} \le \frac{-x + a_1 + a_2 + a_3 + \dots + a_n}{n-1} = \frac{-x + nx}{n-1} = x.$$
This means the new score is at most $x$.

Claim 2. If there is a subset, say $S_1$, with average at most $x$ and having at least two elements, then it is possible to split it into two subsets so that the score does not increase.
Proof. Let $m$ be the smallest element in $S_1$. We split $S_1$ into $\{m\}$ and $S_1 - \{m\}$. The new averages of these two sets are $m$ and $\frac{|S_1|a_1 - m}{|S_1| - 1}$ respectively. Now, we have
$$\begin{align*} \frac{1}{n+1} \left( m + \frac{|S_1|a_1 - m}{|S_1| - 1} + a_2 + a_3 + \dots + a_n \right) \\ &= \frac{1}{n+1} \left( m + \frac{a_1 - m}{|S_1| - 1} + a_1 + a_2 + \dots + a_n \right) \\ &\le \frac{m + (a_1 - m) + nx}{n+1} \le \frac{x + nx}{n+1} = x. \end{align*}$$
This means the new score is at most $x$.

Using the above claims, we merge any two subsets with scores above average and subdivide any subset with score below average into singletons. Also, if there are subsets $\{a\}$ and $B \cup \{b\}$ for which $a \ge b$, then we can swap $a$ and $b$ to get the subsets $\{b\}$ and $B \cup \{a\}$ so that the average decreases, since
$$a + \frac{s+b}{|B|+1} > b + \frac{s+a}{|B|+1}$$
if $|B| \ge 1$.
Therefore, the optimal case must be of the form
$$\{1\}, \{2\}, \{3\}, \dots, \{k\}, \{k+1, k+2, \dots, 100\}$$
for some $k \le 100$. The score is
$$\frac{1}{k+1} \left( \frac{k(k+1)}{2} + \frac{k+101}{2} \right) = \frac{k+1}{2} + \frac{100}{2(k+1)} \ge 2 \sqrt{\frac{k+1}{2} \cdot \frac{100}{2(k+1)}} = 10$$
by the AM-GM inequality. In other words, the minimum score is $10$, which is attained when $k = 9$.