Let $C$ be the center of the base of the cone, $A$ be the center of one of the spheres, $B$ be the point where that sphere is tangent to the base of the cone, $D$ be the point such that $\overline{CD}$ is a radius of the base of the cone containing $B$, $V$ be the vertex of the cone, and $E$ be the point on $\overline{CV}$ such that $\overline{ED}$ contains $A$. Let the radius of the sphere be $r = AB$.

Because the three spheres are mutually tangent, their centers are at the vertices of an equilateral triangle with side length $2r$. The line $CV$ passes through the centroid of the equilateral triangle, so $BC$ must be $\frac{2}{3}$ the altitude of that triangle, implying that $BC = \frac{2}{3} \cdot r\sqrt{3}$.

The Pythagorean Theorem implies that $DV = \sqrt{CD^2 + CV^2} = 13$.

Observe that $\overline{DE}$ is the angle bisector of $\angle CDV$. By the Angle Bisector Theorem, $\frac{CD}{CE} = \frac{DV}{EV}$. Thus $\frac{5}{CE} = \frac{13}{12-CE}$ and solving gives $CE = \frac{10}{3}$.

Because $\triangle DAB \sim \triangle DEC$, it follows that
$$BD = AB \cdot \frac{CD}{CE} = r \cdot \frac{5}{\left(\frac{10}{3}\right)} = \frac{3}{2}r.$$
Then $BC + BD = CD$ gives
$$\frac{2\sqrt{3}}{3}r + \frac{3}{2}r = 5.$$
Solving for $r$ yields $r = \frac{90-40\sqrt{3}}{11}$.