Saudi Arabian Mathematical Competitions, 2012

If $a_1 < a_2 < \dots < a_n$ are the elements of $A$, then $a_1^2 + a_2^2, a_1^2 + a_3^2, \dots, a_1^2 + a_n^2, a_2^2 + a_n^2, \dots, a_{n-1}^2 + a_n^2$ belong to $A$, and we have
$a_1^2 + a_2^2 < a_1^2 + a_3^2 < \dots < a_1^2 + a_n^2 < a_2^2 + a_n^2 < \dots < a_{n-1}^2 + a_n^2$,
which implies $2n - 3 \le n$. It follows that $A$ has at most 3 elements.
If $n=3$ and $A = \{a, b, c\}$ with $a < b < c$, then $a^2 < b^2 < c^2$, hence $a^2 + b^2 < a^2 + c^2 < b^2 + c^2$.
Since $A = \{a, b, c\} = \{a^2 + b^2, a^2 + c^2, b^2 + c^2\}$, it follows
$$a^2 + b^2 = a, \quad a^2 + c^2 = b, \quad b^2 + c^2 = c. \quad (1)$$
From the first and the last relation in (1) we get $a^2 - c^2 = a - c$, hence $a + c = 1$. Substituting in the second relation of (1) it follows
$$b = a^2 + (1 - a)^2 = 2a^2 - 2a + 1 = 2(a^2 - a) + 1 = -2b^2 + 1,$$
therefore $(2b - 1)(b + 1) = 0$, hence $b = \frac{1}{2}$. Then $a^2 - a + \frac{1}{4} = 0$
leads to $a = \frac{1}{2}$, which contradicts $a < b$.