Mathematical competitions in Croatia

By considering the given expression modulo $8$, we get
$$(a + 3b)(5a + 7b) \equiv 5a^2 + 22ab + 21b^2 \equiv 5a^2 - 10ab + 5b^2 \equiv 5(a - b)^2 \pmod{8}.$$
Let us assume the contrary, that $(a + 3b)(5a + 7b) = k^2$ for some positive integer $k$.
Then
$$k^2 \equiv 5(a - b)^2 \pmod{8}.$$
Since $a$ and $b$ are of different parity, $a - b$ is odd, so that $(a - b)^2$ gives remainder $1$ when divided by $8$.
This implies $k^2 \equiv 5 \pmod{8}$. However, $5$ is not a quadratic residue modulo $8$.
We have arrived at a contradiction, so we conclude that the initial assumption was false. Therefore, $(a + 3b)(5a + 7b)$ is not a perfect square.