If $(ab)^2 - 4(a+b) = x^2$ with positive integers $a, b$ and an integer $x \ge 0$, we have $x < ab$. As $(ab)^2 - (ab-1)^2 = 2ab - 1$ is odd, we even have $x \le ab - 2$. This implies $(ab)^2 - 4(a+b) \le (ab-2)^2 = (ab)^2 - 4ab + 4$, from which we obtain
$$ab \le a + b + 1. \qquad (5)$$
After swapping $a$ and $b$ if necessary, we may assume $a \le b$. If $a \ge 3$, we get $ab \ge 3b \ge a+b+b \ge a+b+1$ in contradiction to (5). Hence $a = 2$ or $a = 1$. If $a = 1$, we have $b^2 - 4(b+1) = x^2$, which is equivalent to $(b-2-x)(b-2+x) = 8$. Because $(b-2-x)+(b-2+x) = 2b-4$ is even and $b-2-x \le b-2+x$, the only possibility is $b-2-x = 2$ and $b-2+x = 4$. This yields $(a, b) = (1, 5)$ as the only possible solution with $1 = a \le b$.

If $a = 2$, we have $4b^2 - 4(b+2) = x^2$, equivalently $(2b-1-x)(2b-1+x) = 9$. Here we have two possibilities. Either $2b-1-x = 2b-1+x = 3$ or $2b-1-x = 1, 2b-1+x = 9$. In the first case we obtain $b=2$ and in the second $b=3$. So we have shown that $(a,b) = (2,2)$ and $(a,b) = (2,3)$ are the only possible solutions with $2 = a \le b$.