Irish Mathematical Olympiad

(i) For a root of unity $z = p + qi$, with $p$ and $q$ real and $z^k = 1$, we have $|z|^k = |z^k| = 1$, i.e. $p^2+q^2=1$. So, if $(a+bi)/\sqrt{n} \in S_n$, for positive integers $a$ and $b$, we have $a^2+b^2=n$.
Since $a$ and $b$ are positive integers, $a < \sqrt{n}$ and, once $a$ is known, $b$ is also known, so $S_n$ has fewer than $\sqrt{n}$ distinct elements, as asserted.

(ii) Suppose to the contrary that $z = (a+bi)/\sqrt{5}$ is a root of unity, where $a$ and $b$ are positive integers. Then $a^2+b^2=5$ and we have two possibilities: either $a=1$ and $b=2$ or $a=2$ and $b=1$.
First note that $2+i = i(1+2i)$ and that $z$ is a root of unity iff $\bar{z}$ is a root of unity. Hence it is sufficient to consider the case
$$z = \frac{1+2i}{\sqrt{5}} = \frac{1}{\sqrt{5}} + \frac{2}{\sqrt{5}}i.$$
Suppose $z$ is a root of unity, say $z^k = 1$, where $k$ is a positive integer. We can write $z = \cos \theta + i \sin \theta$, and obtain $\cos(k\theta) = 1$, $\sin(k\theta) = 0$, so $k\theta = 2m\pi$, for some integer $m$, and if $l$ is any integer, then $\cos(l\theta) = \cos(h\pi/k)$, for some non-negative integer $h < k$.
In particular, there are only finitely many possible values for $\cos(l\theta)$ as $l$ runs through the set of integers.
We claim:
$$\cos(2^r \theta) = \frac{u_r}{5^{2r-1}} \quad \text{for } r = 1, 2, 3, \dots$$
where $u_r$ is an integer not divisible by $5$.
We prove the claim by induction on $r$. When $r=1$, $\cos(2^r\theta) = \cos(2\theta) = 2\cos^2\theta - 1 = 2(\frac{1}{\sqrt{5}})^2 - 1 = -\frac{3}{5}$, and the claim holds.
$$2 \cos^2 \theta - 1 = 2 \left( \frac{1}{\sqrt{5}} \right)^2 - 1 = -\frac{3}{5}$$
Assume that $r \ge 1$ is an integer and that the claim holds for $r$. This means that $\cos(2^r\theta) = \frac{u_r}{5^{2r-1}}$, with $u_r$ not divisible by $5$, and thus
$$\cos(2^{r+1}\theta) = 2\cos^2(2^r\theta) - 1 = \frac{2u_r^2}{(5^{2r-1})^2} - 1 = \frac{2u_r^2 - 5^{2r}}{5^{2r}},$$
and $u_{r+1} = 2u_r^2 - 5^{2r}$ is an integer not divisible by $5$. Thus the claim holds for $r+1$. Hence the claim is proved for all integers $r \ge 1$.
It follows that all the values of $\cos(2^r\theta)$, $r = 1, 2, 3, \dots$ are distinct, since, when written as a fraction in reduced form, all the denominators are distinct. But this contradicts what we have shown above, namely that there are only finitely many possible values for $\cos(l\theta)$ as $l$ runs through the set of integers. Hence $S_5$ is empty, as required.