HMMT November, 2022

Solution:
There are no such pairs when $b=57$, so we may only consider pairs with $1 \leq a < b \leq 56$. The key idea is that unless $a^{2} \bmod 57 = b^{2} \bmod 57$, $(a, b)$ can be paired with $(57-b, 57-a)$ and exactly one of them satisfies $a^{2} \bmod 57 < b^{2} \bmod 57$. Hence if $X$ is the number of pairs $(a, b)$ with $1 \leq a < b \leq 56$ and $a^{2} \equiv b^{2} \pmod{57}$, then the answer is
$$\frac{1}{2}\left(\binom{56}{2} - X\right)$$
To count $X$, let's first count the number of pairs $(a, b)$ with $1 \leq a, b \leq 57$ and $a^{2} \equiv b^{2} \pmod{57}$. By the Chinese Remainder Theorem, the condition is equivalent to $(a-b)(a+b) \equiv 0 \pmod{3}$ and $(a-b)(a+b) \equiv 0 \pmod{19}$. There are $2 \cdot 3 - 1 = 5$ pairs of residues modulo $3$ where $(a-b)(a+b) \equiv 0 \pmod{3}$, namely $(0,0), (1,1), (2,2), (1,2), (2,1)$. Similarly, there are $2 \cdot 19 - 1 = 37$ pairs of residues modulo $19$ where $(a-b)(a+b) \equiv 0 \pmod{19}$. By the Chinese Remainder Theorem, each choice of residues modulo $3$ for $a$ and $b$ and residues modulo $19$ for $a$ and $b$ corresponds to unique residues modulo $57$ for $a$ and $b$. It follows that there are $5 \cdot 37 = 185$ such pairs. To get the value of $X$, we need to subtract the $57$ pairs where $a = b$ and divide by $2$ for the pairs with $a > b$, for a value of $X = \frac{1}{2}(185 - 57) = 64$.
Therefore the final answer is
$$\frac{1}{2}\left(\binom{56}{2} - 64\right) = 738$$