Let $n$ be the number of students.

Initially, the $960$ tokens are distributed equally, so each student gets $\dfrac{960}{n}$ tokens.

When the teacher arrives, each student gives him $4$ tokens, so the teacher receives $4n$ tokens in total. Now there are $n+1$ people (the $n$ students and the teacher), and the $960$ tokens are distributed among them equally, so each person gets $\dfrac{960}{n+1}$ tokens.

After giving away $4$ tokens, each student has:
$$\frac{960}{n} - 4$$
tokens.

The teacher has $4n$ tokens.

Since now everyone has the same number of tokens:
$$\frac{960}{n} - 4 = \frac{960}{n+1}$$

Multiply both sides by $n(n+1)$:
$$(n+1) \cdot 960 - 4n(n+1) = n \cdot 960$$

Expand:
$$960n + 960 - 4n^2 - 4n = 960n$$

Subtract $960n$ from both sides:
$$960 - 4n^2 - 4n = 0$$

Bring all terms to one side:
$$4n^2 + 4n - 960 = 0$$

Divide both sides by $4$:
$$n^2 + n - 240 = 0$$

Solve the quadratic equation:
$$n = \frac{ -1 \pm \sqrt{1 + 4 \cdot 240} }{2 } = \frac{ -1 \pm \sqrt{961} }{2 } = \frac{ -1 \pm 31 }{2 }$$

So $n = \frac{30}{2} = 15$ or $n = \frac{-32}{2} = -16$.

Since $n$ must be positive, $n = 15$.

Answer: $15$ students were playing the game.