Serbian Mathematical Olympiad

Solution:

Let $b^{2}=c a$. The conditions of the problem give $b^{2}=c a \mid a^{4}$ and $a+1 \mid c a+1$, and this is equivalent to
$$c \mid a^{3} \quad \text{and} \quad a+1 \mid c-1.$$
Let $c=d(a+1)+1,\ d \in \mathbb{N}_{0}$. Since $a^{3} \equiv -1 \pmod{a+1}$, we have $\frac{a^{3}}{c} \equiv -1 \pmod{a+1}$, i.e. $\frac{a^{3}}{c}=e(a+1)-1$ for some $e \in \mathbb{N}$. It follows that $a^{3}=(d(a+1)+1)(e(a+1)-1)$, which after multiplying out and cancelling $a+1$ becomes $a^{2}-a+1=d e(a+1)+(e-d)$. From here we have $e-d \equiv a^{2}-a+1 \equiv 3 \pmod{a+1}$, hence
$$e-d=k(a+1)+3 \quad \text{and} \quad d e=a-2-k \quad (k \in \mathbb{Z}).$$
We distinguish the following cases:

(1) $k \notin\{-1,0\}$. In this case (*) implies $d e<|e-d|-1$, which is possible only for $d=0$. Now $c=1$ and $b^{2}=a$, hence $(a, b)=\left(t^{2}, t\right)$.

(2) $k=-1$. From (*) we obtain $a=d+1$. Now $c=a^{2}$ and $b^{2}=a^{3}$, hence $(a, b)=\left(t^{2}, t^{3}\right)$.

(3) $k=0$. From (*) we obtain $a=d^{2}+3 d+2$. Now $c=d(a+1)+1=(d+1)^{3}$ and $b^{2}=c a=(d+1)^{4}(d+2)$. It follows that $d+2=t^{2}$ for some $t \in \mathbb{N}$, which gives $(a, b)=\left(t^{2}(t^{2}-1), t(t^{2}-1)^{2}\right),\ t \geq 2$.

Answer: These are the pairs $(a, b)$ of the form $\left(t^{2}, t\right),\ \left(t^{2}, t^{3}\right)$ and $\left(t^{2}(t^{2}-1), t(t^{2}-1)^{2}\right)$, where $t \in \mathbb{N}$.