Answer (C): More generally, let $a = BC$, $b = AC$, and $c = AB$ where $a < b$; then $c = \sqrt{a^2 + b^2}$. Because $AB$ is a chord of both circles, their centers $O$ and $P$ must lie on the perpendicular bisector of $AB$. Letting $M$ be the midpoint of $AB$, observe that $\triangle OMB$ is a right triangle, and radius $OB$ is perpendicular to tangent $BC$, so it is parallel to $AC$. Thus $\angle OBM = \angle BAC$, so $\triangle OMB$ is similar to $\triangle BCA$, and $OM = BC \cdot \frac{MB}{AC} = \frac{ac}{2b}$.

Likewise, $\triangle AMP$ is similar to $\triangle BCA$, so $MP = AC \cdot \frac{MA}{BC} = \frac{bc}{2a}$. Hence
$$OP = MP - OM = \frac{c}{2} \left( \frac{b}{a} - \frac{a}{b} \right) = \frac{c}{2} \left( \frac{b^2 - a^2}{ab} \right),$$
which for the given $\triangle ABC$ is equal to
$$\frac{10}{2} \left( \frac{8^2 - 6^2}{6 \cdot 8} \right) = \frac{35}{12}.$$

OR

Let $C$ be the origin of a coordinate system with $B = (0, 6)$ and $A = (8, 0)$. The circle centered at point $O$ is tangent to $BC$ at $B$, so $O = (x, 6)$ for some $x$. Similarly, $P = (8, y)$ for some $y$. Points $A$ and $B$ lie on the circle centered at $O$, so $(8-x)^2 + 6^2 = x^2 + 0^2$. This simplifies to $16x = 100$, so $x = \frac{25}{4}$. Points $A$ and $B$ also lie on the circle centered at $P$, so $0^2 + y^2 = 8^2 + (y-6)^2$. This simplifies to $12y = 100$, so $y = \frac{25}{3}$. It follows that
$$OP^2 = (8-x)^2 + (y-6)^2 = \left(\frac{7}{4}\right)^2 + \left(\frac{7}{3}\right)^2 = \frac{49 \cdot 25}{144},$$
whence $OP = \frac{35}{12}$.