We can take half of a cross section of the sphere, as such:

Notice that we chose a cross section where one of the spheres was tangent to the lateral surface of the cone at $D$.
To evaluate $r$, we will find $AE$ and $EC$ in terms of $r$; we also know that $AE+EC = 5$, so with this, we can solve $r$. Firstly, to find $EC$, we can take a bird's eye view of the cone:

Note that $C$ is the centroid of equilateral triangle $EXY$. Also, since all of the medians of an equilateral triangle are also altitudes, we want to find two-thirds of the altitude from $E$ to $XY$; this is because medians cut each other into a $2$ to $1$ ratio. This equilateral triangle has a side length of $2r$, therefore it has an altitude of length $r \sqrt{3}$; two thirds of this is $\frac{2r \sqrt{3}}{3}$, so $EC = \frac{2r \sqrt{3}}{3}.$

To evaluate $AE$ in terms of $r$, we will extend $\overline{OE}$ past point $O$ to $\overline{AB}$ at point $F$.$\triangle AEF$ is similar to $\triangle ACB$. Also, $AO$ is the angle bisector of $\angle EAB$. Therefore, by the angle bisector theorem, $\frac{OE}{OF} = \frac{AE}{AF} = \frac{5}{13}$. Also, $OE = r$, so $\frac{r}{OF} = \frac{5}{13}$, so $OF = \frac{13r}{5}$. This means that$$AE = \frac{5 \cdot EF}{12} = \frac{5 \cdot (OE + OF)}{12} = \frac{5 \cdot (r + \frac{13r}{5})}{12} = \frac{18r}{12} = \frac{3r}{2}.$$
We have that $EC = \frac{2r \sqrt{3}}{3}$ and that $AE = \frac{3r}{2}$, so $AC = EC + AE = \frac{2r \sqrt{3}}{3} + \frac{3r}{2} = \frac{4r \sqrt{3} + 9r}{6}$. We also were given that $AC = 5$. Therefore, we have
$$\frac{4r \sqrt{3} + 9r}{6} = 5.$$
This is a simple linear equation in terms of $r$. We can solve for $r$ to get $r = \boxed{\textbf{(B)}\ \frac{90-40\sqrt{3}}{11}}.$
~ihatemath123