, 2011

Consider a general triangle $ABC$. Suppose we have a circle that touches the lines $AB$ and $AC$. Since it cannot also pass through the point $A$, we may suppose it passes through the point $C$. The centre of the circle will then lie either on the internal, or the external, bisector of the angle at $A$.

Assume the centre of the circle lies on the internal bisector. Then its radius is
$$r = b \tan \frac{A}{2} = 2R \sin B \tan \frac{A}{2},$$
where $R$ denotes the circumradius. The maximal radius is obtained when $A \ge B \ge C$ (the expression $\frac{\sin x}{\tan \frac{x}{2}} = 2 \cos^2 \frac{x}{2}$ is strictly decreasing for $0 \le x \le 180^\circ$).

Assume now the centre of the circle lies on the external bisector. Then its radius is
$$s = b \tan \frac{B+C}{2} = \frac{2R \sin B}{\tan \frac{A}{2}}.$$
The maximal radius is obtained when $B \ge C \ge A$.

For the triangle at hand, $r$ is maximized by $b = 4$ and $A = 90^\circ$, which gives $r = 4$, and $s$ by $b = 5$ and $A$ the angle opposite the side of length $3$. Then $\tan A = \frac{3}{4}$, $\tan \frac{A}{2} = \frac{1}{3}$, which produces the greatest radius $s = 15$, which is thus the answer.