Let the quadrilateral $EFGH$ be inscribed in the triangle $ABC$ so that $E$ and $F$ lie on $BC$ and $G$ lies on $AC$ and $H$ lies on $AB$. Let us denote the side-lengths of the triangle $ABC$ by $a$, $b$ and $c$ and let $h$ denote the length of the height drawn from $A$ to $BC$. We put $\overline{AH} = x$, $\overline{EF} = u$ and $\overline{FG} = v$.

From $\triangle AHG \sim \triangle ABC$ we have $u = \frac{a x}{c}$, from $\triangle BEH \sim \triangle BVA$ we get $v = \frac{h(c - x)}{c}$. If $l$ denotes the length of the diagonal of $EFGH$, we get
$$l^2 = \frac{a^2 x^2}{c^2} + \frac{h^2 (c - x)^2}{c^2}.$$
The smallest value of the parabola
$$f(x) = \frac{a^2 x^2}{c^2} + \frac{h^2 (c - x)^2}{c^2}$$
is $\frac{a^2 h^2}{a^2 + h^2}$ and it is attained when $x = \frac{h^2 c}{a^2 + h^2}$. Let us note that
$$\frac{a^2 h^2}{a^2 + h^2} = \frac{4P^2}{a^2 + \frac{4P^2}{a^2}},$$
where $P$ is the area of the triangle. Now if we do the same when the rectangle has two neighbouring vertices lying on the side $AC$, we get $\frac{4P^2}{b^2 + \frac{4P^2}{b^2}}$ for the smallest possible length of the diagonal.

We have
$$\frac{a^2 + \frac{4P^2}{a^2} - b^2 - \frac{4P^2}{b^2}}{a^2 + \frac{4P^2}{b^2}} = (a^2 - b^2)\left(1 - \frac{4P^2}{a^2 b^2}\right).$$
Since $ab \ge 2P$ the last expression is greater or equal to $0$ if and only if $a \ge b$. In other words, the smallest value of $l$ is obtained when the rectangle has two neighbouring vertices lying on the longest side of the triangle. Let $a$ be the longest side. We already saw that the smallest value of $l$ is
$$\frac{2P}{\sqrt{a^2 + \frac{4P^2}{a^2}}}$$
and it is obtained when $\overline{AH} = x = \frac{h^2 c}{a^2 + h^2} = \frac{4P^2 c}{a^4 + 4P^2}$. Let us note that the area is known and it can be expressed through the side-lengths of the triangle by e.g. Heron's formula.