, 2015

Solution:

Answer: $\sqrt{\frac{25}{9}}$

Without loss of generality, we assume that $c \geq b \geq a$. We see that $3c \geq a + b + c = 10$. Therefore, $c \geq \frac{10}{3}$.

Since
$$\begin{aligned} 0 &\leq (a-b)^2 \\ &= (a+b)^2 - 4ab \\ &= (10-c)^2 - 4(25 - c(a+b)) \\ &= (10-c)^2 - 4(25 - c(10-c)) \\ &= c(20 - 3c) \end{aligned}$$
we obtain $c \leq \frac{20}{3}$.

Consider $m = \min \{ ab, bc, ca \} = ab$, as $bc \geq ca \geq ab$. We compute
$$ab = 25 - c(a+b) = 25 - c(10 - c) = (c - 5)^2.$$
Since $\frac{10}{3} \leq c \leq \frac{20}{3}$, we get that $ab \leq \frac{25}{9}$. Therefore, $m \leq \frac{25}{9}$ in all cases and the equality can be obtained when $(a, b, c) = \left(\frac{5}{3}, \frac{5}{3}, \frac{20}{3}\right)$.