Iranian Mathematical Olympiad

Reformulate the original expression as $x^2 - 1 = (p+1)^2(y^2 - 1)$. $x^2 - 1 = ((p+1)y)^2 - (p+1)^2 \Rightarrow (p+1)^2 - 1 = ((p+1)y)^2 - x^2$. $p^2 + 2p + 1 - 1 = ((p+1)y - x)((p+1)y + x)$. So, $p(p+2) = ((p+1)y - x)((p+1)y + x)$. Let $A = (p+1)y - x$ and $B = (p+1)y + x$. Then $AB = p(p+2)$. Since $x, y > 1$, $(p+1)y \ge 2(p+1) = 2p+2$. $x \ge 2$. $B = (p+1)y + x \ge 2(p+1) + 2 = 2p+4$. Also, $B - A = 2x > 0 \Rightarrow B > A$. $B + A = 2(p+1)y$. Since $p$ is a prime, it must either divide $A$ or $B$.

Case I. $p|A$. So $A = kp$ for some integer $k$. Then $B = (p+2)/k$. If $p$ divides the first bracket, i.e., (A), then $(p+1)y+x|p+2$. This implies $B|p+2$. Since $B \ge 2p+4$, we have $2p+4 \le p+2 \Rightarrow p \le -2$, which is impossible.

Case II. $p|B$. So $(p+1)y+x = kp$ for some integer $k$. And $(p+1)y-x = (p+2)/k$. We have $y \equiv -x \pmod p$. Let $x = kp - y$ for some integer $k$. Substitute this into $A \cdot B = p(p+2)$: $B = (p+1)y+x = (p+1)y + kp - y = py + kp = p(y+k)$. $A = (p+1)y - x = (p+1)y - (kp - y) = (p+2)y - kp$. So,
$$((p+2)y - kp) \cdot p(y + k) = p(p + 2) \implies ((p+2)y - kp)(y + k) = p + 2$$
If $y > k$, then $(p(y - k) + 2y)(y + k) > p + 2$ which is a contradiction. Thus $y \le k$. Since $y + k$ is a factor of $p + 2$, $y + k \le p + 2$, so we have $2y \le p + 2$. If $y - k \le -2$ we have
$$p(y - k) + 2y \le -2p + 2y < 0$$
which is a contradiction. The only remaining possibility is $y - k = -1$ (i.e. $k = y+1$) or $y - k = 0$ (i.e. $k = y$). If $k = y$: $((p+2)y - py)(y+y) = p+2 \implies (2y)(2y) = p+2 \implies 4y^2 = p+2$. If $p = 2$, $4y^2 = 4 \implies y^2 = 1 \implies y = 1$. Rejected. If $p$ is odd it follows that $p+2$ is also odd since $4y^2$ is even then $p+2$ cannot be odd. Yielding $4y^2 = p+2$ has no solution for odd $p$. If $k = y+1$: $((p+2)y - p(y+1))(y+y+1) = p+2 \implies (py+2y - py - p)(2y+1) = p+2$. $(2y-p)(2y+1) = p+2$. $4y^2+2y-2py-p = p+2 \implies 4y^2+2(1-p)y-(2p+2) = 0$. Solving $4y^2 + 2(1-p)y - (2p+2) = 0$, with respect to $y$, we have
$$y = \frac{p+1}{2}.$$
This requires $p$ to be odd. Then
$$k = y + 1 = \frac{p+1}{2} + 1 = \frac{p+3}{2}$$
And
$$x = kp - y = \left(\frac{p+3}{2}\right) p - \frac{p+1}{2} = \frac{p^2 + 3p - p - 1}{2} = \frac{p^2 + 2p - 1}{2}$$
The solutions are $(x, y) = \left(\frac{p^2+2p-1}{2}, \frac{p+1}{2}\right)$ for odd primes $p$.