, 2020

Solution:

Let $O$ be the common center of the two spheres. Projecting $O$ onto each face of the tetrahedron will divide it into three isosceles triangles. Unfolding the tetrahedron into its net, the reflection of any of these triangles about a side of the tetrahedron will coincide with another one of these triangles. Using this property, we can see that each of the faces is broken up into the same three triangles. It follows that the tetrahedron is isosceles, i.e. $AB = CD$, $AC = BD$, and $AD = BC$.

Let $P$ be the projection of $O$ onto $ABC$ and $x = BC$. By the Pythagorean Theorem on triangle $POA$, $P$ has distance $\sqrt{R^2 - r^2} = r\sqrt{15}$ from $A$, $B$, and $C$. Using the area-circumcenter formula, we compute
$$[ABC] = \frac{AB \cdot AC \cdot BC}{4PA} = \frac{x}{4r\sqrt{15}}$$
However, by breaking up the volume of the tetrahedron into the four tetrahedra $OABC$, $OABD$, $OACD$, $OBCD$, we can write $[ABC] = \frac{V}{\frac{4}{3}r}$, where $V = [ABCD]$. Comparing these two expressions for $[ABC]$, we get $x = 3\sqrt{15}V$.

Using the formula for the volume of an isosceles tetrahedron (or some manual calculations), we can compute $V = x^2 \sqrt{\frac{1}{72}(2 - x^2)}$. Substituting into the previous equation (and taking the solution which is $\geq 1$), we eventually get $x^2 = 1 + \sqrt{\frac{7}{15}}$.