HMMT February, 2018

Solution:
Consider a right triangle $A O C$ with right angle at $O$, $A O = 16$ and $C O = x$. Moreover, let $B$ be on $A O$ such that $B O = 9$. Then $\tan^{-1} \frac{x}{9} = \angle C B O$ and $\tan^{-1} \frac{x}{16} = \angle C A O$, so their difference is equal to $\angle A C B$.

Note that the locus of all possible points $C$ given the value of $\angle A C B$ is part of a circle that passes through $A$ and $B$, and if we want to maximize this angle then we need to make this circle as small as possible. This happens when $O C$ is tangent to the circumcircle of $A B C$, so $O C^2 = O A \cdot O B = 144 = 12^2$, thus $x = 12$, and it suffices to compute $\sin (\alpha - \beta)$ where $\sin \alpha = \cos \beta = \frac{4}{5}$ and $\cos \alpha = \sin \beta = \frac{3}{5}$.

By angle subtraction formula we get
$$\sin (\alpha - \beta) = \left(\frac{4}{5}\right)^2 - \left(\frac{3}{5}\right)^2 = \frac{7}{25}.$$