Harvard-MIT Mathematics Tournament, Team Round B

Solution:

Let $A, B, C, P$ be the centers of the circles with radii $1, 2, 3$, and $r$, respectively. Then, $ABC$ is a $3$-$4$-$5$ right triangle. Using the law of cosines in $\triangle PAB$ yields
$$\cos \angle PAB = \frac{3^2 + (1 + r)^2 - (2 + r)^2}{2 \cdot 3 \cdot (1 + r)} = \frac{3 - r}{3(1 + r)}$$
Similarly,
$$\cos \angle PAC = \frac{4^2 + (1 + r)^2 - (3 + r)^2}{2 \cdot 4 \cdot (1 + r)} = \frac{2 - r}{2(1 + r)}$$
We can now use the equation $(\cos \angle PAB)^2 + (\cos \angle PAC)^2 = 1$, which yields $0 = 23r^2 + 132r - 36 = (23r - 6)(r + 6)$, or $r = 6/23$.