HMMT February February 16, 2019

Solution:
We claim that the circle in question is the circumcircle of the anticomplementary triangle of $ABC$, the triangle for which $ABC$ is the medial triangle.

Let $A'B'C'$ be the anticomplementary triangle of $ABC$, such that $A$ is the midpoint of $B'C'$, $B$ is the midpoint of $A'C'$, and $C$ is the midpoint of $A'B'$. Denote by $\omega$ the circumcircle of $A'B'C'$. Denote by $\omega_A$ the circumcircle of $BHC$, and similarly define $\omega_B$, $\omega_C$.

Since $\angle BA'C = \angle BAC = 180^\circ - \angle BHC$, we have that $\omega_A$ passes through $A'$. Thus, $\omega_A$ can be redefined as the circumcircle of $A'BC$. Since triangle $A'B'C'$ is triangle $A'BC$ dilated by a factor of $2$ from point $A'$, $\omega$ is $\omega_A$ dilated by a factor of $2$ from point $A'$. Thus, circles $\omega$ and $\omega_A$ are tangent at $A'$.

By a similar logic, $\omega$ is also tangent to $\omega_B$ and $\omega_C$. Therefore, the circumcircle of the anticomplementary triangle of $ABC$ is indeed the circle that the question is asking for.

Using the formula $R = \frac{abc}{4A}$, we can find that the circumradius of triangle $ABC$ is $\frac{65}{8}$. The circumradius of the anticomplementary triangle is double of that, so the answer is $\frac{65}{4}$.

Solution:
It is well-known that the circumcircle of $AHB$ is the reflection of the circumcircle of $ABC$ over $AB$. In particular, the circumcircle of $AHB$ has radius equal to the circumradius $R = \frac{65}{8}$. Similarly, the circumcircles of $BHC$ and $CHA$ have radii $R$. Since $H$ lies on all three circles (in the question), the circle centered at $H$ with radius $2R = \frac{65}{4}$ is tangent to each circle at the antipode of $H$ in that circle.