Team Selection Test for JBMO

a. The answer: permutations of $(2, 3, 11)$.
Let $p+q+r = x^2$ and $pq+qr+rp+3 = y^2$ where $x$, $y$ are integers. Let us show that one of the primes $p$, $q$, $r$ is $2$. If all primes $p$, $q$, $r$ are odd all possibilities up to permutations are: $(p, q, r) = (1, 1, 1)$, $(1, 1, 3)$, $(1, 3, 3)$, $(3, 3, 3)$ (mod $4$). We get a contradiction in the cases $(1, 1, 1)$, $(1, 3, 3)$ since $x^2 = p+q+r \equiv 3 \pmod{4}$ and in the cases $(1, 1, 3)$, $(3, 3, 3)$ since $y^2 - 3 = pq + qr + rp \equiv 3 \pmod{4}$.
Therefore, at least one of $p$, $q$, $r$ is equal to $2$. W.l.o.g. $p=2$ and $q \le r$. Then $q+r=x^2-2$, $qr=y^2-2x^2+1$.
Now if $3 \mid y$, then $(q+2)(r+2) = y^2+1 \equiv 1 \pmod{3}$. Thus, either $q \equiv r \equiv 2 \pmod{3}$ or $q \equiv r \equiv 0 \pmod{3}$. But for $q \equiv r \equiv 0 \pmod{3}$ we get a contradiction: $x^2 - 2 \equiv 0 \pmod{3}$. For $q \equiv r \equiv 2 \pmod{3}$ we get $x^2 - 2 \equiv 1 \pmod{3}$ and $3 \mid x$, but by assumption $3 \nmid x$. Thus, $3 \mid y$ is not possible. Now since $3 \nmid x$ we get $x^2 \equiv y^2 \equiv 1 \pmod{3}$ and consequently $qr = y^2 - 2x^2 + 1 \equiv 0 \pmod{3}$. Thus, $q=3$. Now $r = x^2 - 5$ and $3r = y^2 - 2x^2 + 1$. Therefore $5r = y^2 - 9 = (y-3)(y+3)$. For $r=2, 3, 5$ $x$ is not an integer number. Therefore, $r > 5$. Since $y-3=1$ yields no solution $y-3=5$, $r = y+3$

and $r = 11$. For $x = 4$, $y = 8$ we get $(p, q, r) = (2, 3, 11)$.

b. $(p, q, r) = (2, 11, 23)$ satisfies the conditions for $(x, y) = (6, 18)$.