Irish Mathematical Olympiad

Let $DEFG$ be a square inscribed in $\triangle ABC$ as described in the problem, let $AH$ be the altitude at $A$ and $h_a = |AH|$ its length. By $a$, $b$, $c$ we denote the lengths of the sides of $\triangle ABC$ in the usual way. Let $x_a$ denote the side length of the square if one of its sides lies on the side $BC$ of the triangle (see figure below). Similarly, $x_b$ and $x_c$ are the side lengths of the rectangles in the two other cases.

From similar triangles we obtain
$$\frac{|BE|}{x_a} = \frac{|BH|}{h_a} \quad \text{and} \quad \frac{|FC|}{x_a} = \frac{|HC|}{h_a}$$
This implies
$$a = |BE| + x_a + |FC| = \frac{x_a}{h_a} (|BH| + |HC|) + x_a = \frac{a x_a}{h_a} + x_a = \left(1 + \frac{a}{h_a}\right) x_a$$
and so, letting $\Delta = \frac{1}{2} a h_a$ denote the area of $\triangle ABC$,
$$x_a = \frac{a h_a}{a + h_a} = \frac{2\Delta}{a + h_a} = \frac{2a\Delta}{a^2 + 2\Delta}$$
$$x_b = \frac{2b\Delta}{b^2 + 2\Delta}$$
Now we prove that $b > a$ implies $x_a > x_b$. Because $0 < \gamma = \angle ACB < 180^\circ$ (actually $\gamma < 90^\circ$), we have $0 < \sin(\gamma) < 1$. If we assume $b > a$, we obtain $b - a > (b - a) \sin(\gamma)$ from which we get the following inequalities
$$\begin{aligned} \frac{1}{a + b \sin(\gamma)} &> \frac{1}{b + a \sin(\gamma)} \\ \frac{2a\Delta}{a^2 + 2\Delta} &> \frac{2b\Delta}{b^2 + 2\Delta} = x_b. \end{aligned}$$
This shows that $b > a$ implies $x_a > x_b$, hence the square based on the smallest side of the triangle $ABC$ is the largest. The calculation also shows that $\gamma = 90^\circ$ implies $x_a = x_b$. In this case $F = C$.