Solution:

$\boxed{\dfrac{144}{49}}$

Let the side lengths of $\mathcal{P}$ and $\mathcal{Q}$ be $a$ and $b$, respectively. Label two of the vertices of $\mathcal{P}$ as $D$ and $E$ so that $D$ lies on $AB$ and $E$ lies on $AC$, and so that $DE$ is perpendicular to $AB$. The triangle $ADE$ is similar to $ACB$. So $AD = \dfrac{3}{4} a$. Using similar arguments, we find that
$$\frac{3a}{4} + a + b + \frac{4b}{3} = AB = 5$$
so
$$\frac{a}{4} + \frac{b}{3} = \frac{5}{7}$$
Using Cauchy-Schwarz inequality, we get
$$\left(a^2 + b^2\right)\left(\frac{1}{4^2} + \frac{1}{3^2}\right) \geq \left(\frac{a}{4} + \frac{b}{3}\right)^2 = \frac{25}{49}$$
It follows that
$$a^2 + b^2 \geq \frac{144}{49}$$
Equality occurs at $a = \frac{36}{35}$ and $b = \frac{48}{35}$.