Solution:
First, if $S$ contained some $u \neq 0$ with absolute value $<1$, then (by the first condition) every power of $u$ would be in $S$, and $S$ would contain infinitely many different numbers of absolute value $<1$. This is a contradiction.

Now suppose $S$ contains some number $u$ of absolute value $1$ and argument $\theta$. If $\theta$ is not an integer multiple of $\pi / 6$, then $u$ has some power $v$ whose argument lies strictly between $\theta+\pi / 3$ and $\theta+\pi / 2$. Then $u^{2}+v^{2}=u^{2}\left(1+(v / u)^{2}\right)$ has absolute value between $0$ and $1$, since $(v / u)^{2}$ lies on the unit circle with angle strictly between $2 \pi / 3$ and $\pi$. But $u^{2}+v^{2} \in S$, so this is a contradiction.

This shows that the only possible elements of $S$ with absolute value $\leq 1$ are $0$ and the points on the unit circle whose arguments are multiples of $\pi / 6$, giving $N \leq 1+12=13$.

To show that $N=13$ is attainable, we need to show that there exists a possible set $S$ containing all these points. Let $T$ be the set of all numbers of the form $a+b \omega$, where $a, b$ are integers and $\omega$ is a complex cube root of $1$. Since $\omega^{2}=-1-\omega$, $T$ is closed under multiplication and addition. Then, if we let $S$ be the set of numbers $u$ such that $u^{2} \in T$, $S$ has the required properties, and it contains the $13$ complex numbers specified, so we're in business.