Korean Mathematical Olympiad Final Round

Let $p$ be a prime. First suppose that $p$ is an odd prime dividing $a^2 - 2b^2$. Since $a$ and $b$ are relatively prime, $b_1 = 2ab$ is not divisible by $p$. Suppose that there is a positive integer $n$ such that $b_n$ is divisible by $p$. Let $r$ be the smallest positive integer such that $b_r$ is divisible by $p$. Note that $(a - b\sqrt{2})^{2n} = a_n - b_n\sqrt{2}$ because both $a_n$ and $b_n$ are integers, and
$$\begin{cases} a_n = (a^2 + 2b^2)a_{n-1} + 4ab b_{n-1} \\ b_n = 2ab a_{n-1} + (a^2 + 2b^2)b_{n-1}. \end{cases}$$
Then we have
$$\begin{aligned} 0 \equiv b_r &= 2ab((a^2 + 2b^2)a_{r-2} + 4ab b_{r-2} + (a^2 + 2b^2)b_{r-1}) \\ &= 2(a^2 + 2b^2)b_{r-1} - (a^2 - 2b^2)^2 b_{r-2} \equiv 2(a^2 + 2b^2)b_{r-1} \pmod{p}, \end{aligned}$$
which is a contradiction to the assumption. Therefore $b_n$ is not divisible by $p$ for any positive integer $n$.

Now let $p$ be a prime not dividing $a^2 - 2b^2$. Since $b_1 = 2ab$, we may assume that $p$ is odd and $ab$ is not divisible by $p$. Note that
$$b_n = \frac{(a + b\sqrt{2})^{2n} - (a - b\sqrt{2})^{2n}}{\sqrt{2}} = \sum_{k=0,\ k \equiv 1 \pmod{2}}^{2n} 2 \binom{2n}{k} a^{2n-k} b^k 2^{\frac{k-1}{2}}$$
First assume that there is an integer $m$ such that $m^2 \equiv 2 \pmod{p}$. Then we have
$$\begin{aligned} b_n &= \sum_{k=0,\ k \equiv 1 \pmod{2}}^{2n} 2 \binom{2n}{k} a^{2n-k} b^k 2^{\frac{k-1}{2}} \\ &= \sum_{k=0,\ k \equiv 1 \pmod{2}}^{2n} m^2 \binom{2n}{k} a^{2n-k} b^k m^{k-1} \\ &= \frac{(a + bm)^{2n} - (a - bm)^{2n}}{m} \pmod{p}. \end{aligned}$$
Since $(a + bm)(a - bm) = a^2 - m^2 b^2 \equiv a^2 - 2b^2 \not\equiv 0 \pmod{p}$, $b_{\frac{p-1}{2}} \equiv 0 \pmod{p}$ by Fermat's little theorem.

Now suppose that $x^2 \equiv 2 \pmod{p}$ does not have any integer solution, that is, $2$ is a quadratic non-residue modulo $p$. Clearly,
$$\binom{p+1}{k} \equiv 0 \pmod{p} \text{ for any } 2 \leq k \leq p-1.$$
Therefore by Euler's criterion, we have
$$\begin{align*} b_{\frac{p+1}{2}} &= \sum_{k=0,\ k \equiv 1 \pmod{2}}^{p+1} 2 \binom{p+1}{k} a^{p+1-k} b^k 2^{\frac{k-1}{2}} \\ &= 2(p+1)a^p b + (p+1)ab^p 2^{\frac{p+1}{2}} \\ &= 2ab(1 + 2^{\frac{p-1}{2}}) \equiv 0 \pmod{p}. \end{align*}$$
Therefore such a prime is exactly $2$ or relatively prime to $a^2 - 2b^2$. $\square$