, 2024

We will show that there exists only one such triangle and it has sides of length $21$, $28$, $35$ and the two touching circles have radius equal to $5$.

In any right-angled triangle $ABC$ with hypotenuse $AB$ we denote $a = BC$, $b = AC$, $c = AB$. Clearly, two congruent circles with all tangent conditions from the statement exist. Denote them by $k_1(S_1, r)$ and $k_2(S_2, r)$ such that $k_1$ is tangent to $AC$ (and $k_2$ is tangent to $BC$). Moreover, denote the tangent points of these two circles to $AB$ by $T_1$ and $T_2$, respectively, as in the picture. Consider the incircle $k(S, \varrho)$ of the triangle $ABC$ and denote $D$ the tangent point of $k$ with $AB$.

In the first part of the solution we show that the radius $r$ of circles $k_1$ and $k_2$ is determined by the equation
$$r = \frac{c(a + b - c)}{2(a + b)} \qquad (1)$$
In homothety with center $A$ and coefficient $r/\varrho$ which maps $k$ to $k_1$, the point $D$ is mapped to the point $T_1$. Therefore, $|AT_1| = |AD| \cdot r/\varrho$. Analogously, we get the equation $|BT_2| = |BD| \cdot r/\varrho$. Since $T_1T_2S_2S_1$ is a rectangle, we have $|T_1T_2| = |S_1S_2| = 2r$. After plugging everything in the equation $c = |AT_1| + |T_1T_2| + |BT_2|$, by using the relation $|AD| + |BD| = c$ we obtain
$$c = |AD| \cdot \frac{r}{\varrho} + 2r + |BD| \cdot \frac{r}{\varrho} = 2r + (|AD| + |BD|) \cdot \frac{r}{\varrho} = 2r + \frac{cr}{\varrho}.$$

This yields
$$r = \frac{c\varrho}{c + 2\varrho}.$$
After using the well-known formula $\varrho = (a + b - c)/2$, we obtain the desired formula 1. Suppose that the lengths $a, b, c, r$ are integers and put $k = \gcd(a, b, c)$. Then $a = ka_1$, $b = kb_1$ and $c = kc_1$. Moreover, the Pythagorean theorem $a_1^2 + b_1^2 = c_1^2$ implies that the numbers $a_1, b_1, c_1$ are pairwise coprime and $a_1$ and $b_1$ have different parity. The number $c_1$ is odd and the number $a_1 + b_1 - c_1$ is even. After plugging in $a = ka_1$, $b = kb_1$ and $c = kc_1$ to 1 we obtain:
$$r = \frac{kc_1(a_1 + b_1 - c_1)}{2(a_1 + b_1)} = \frac{k}{a_1 + b_1} \cdot c_1 \cdot \frac{a_1 + b_1 - c_1}{2}. \qquad (2)$$
We will show that not only the second fraction, but also the first fraction of the right-hand side of 2 is an integer. Suppose that there is a prime $p$ that divides both $a_1 + b_1$ and $c_1$. Then from the equation $(a_1 + b_1)^2 = c_1^2 + 2a_1b_1$ we have $p \mid a_1b_1$, which is a contradiction since the numbers $a_1, b_1$ and $c_1$ are pairwise coprime. Thus, the number $a_1 + b_1$ is coprime to $c_1$ and also $a_1 + b_1 - c_1$. Since $r$ is an integer, from the equation 2 it follows, that the first fraction is an integer.

By the problem statement, $r$ is a prime. By 2 it is a product of three positive integers. Since $c_1 > a_1 \ge 1$, we necessarily have $c_1 = r$ and the other two fractions in the right-hand side of 2 are equal to 1. Therefore, we have $k = a_1 + b_1$, $a_1 + b_1 - c_1 = 2$ and also $k = a_1 + b_1 = c_1 + 2 = r + 2$. It follows that
$$2a_1b_1 = (a_1 + b_1)^2 - c_1^2 = (r + 2)^2 - r^2 = 4r + 4.$$
After dividing by two we obtain
$$a_1b_1 = 2r + 2 = 2(a_1 + b_1 - 2) + 2 = 2a_1 + 2b_1 - 2.$$
The last equation can be rewritten in the form $(a_1 - 2)(b_1 - 2) = 2$. It easily follows that $\{a_1, b_1\} = \{3, 4\}$, $c_1 = r = a_1 + b_1 - 2 = 5$, and $k = r + 2 = 7$. For right-angled triangle with sides $7 \cdot 3, 7 \cdot 4$ and $7 \cdot 5$ we, indeed, have $r = 5$. This completes the proof.