We adopt the usual convention that $\binom{i}{j} = 0$ unless $0 \le j \le i$.
With this, the binomial coefficients are defined for all integers via the
recursion:

$$\binom{0}{0} = 1, \quad \binom{n}{k} = \binom{n-1}{k} + \binom{n-1}{k-1}$$

It is clear that the circle is oriented and all the students are facing in same direction (clockwise or counterclockwise). We'll call this direction forward.
In any switch consider the taller student to have moved forward and the shorter student to have remained stationary. No backward motion is allowed. With this definition of forward motion, the first two students with heights $h_1$ and $h_2$ are always stationary, while other students potentially move past them.
For $k > 2$, the student with height $h_k$ can never switch places with the student with height $h_{k-1}$, and the former can make at most $k-2$ more forward moves than the latter (when all the students of heights $h_1, \ldots h_{k-2}$ are between $h_k$ and $h_{k-1}$ in the forward direction).
Therefore, if the $(k-1)^{\mathrm{st}}$ student can make $s_{k-1}$ forward steps, the $k^{\mathrm{th}}$ student can make at most $s_{k-1} + k - 2$ steps. With $s_1 = s_2 = 0$ and $s_3 = 0 + (3-2) = 1$, and a constant second difference of $1$, we quickly see that $s_k = \binom{k-1}{2}$.
With $n$ students in all, the total number of steps is therefore at most $\sum_{i=3}^{n}\binom{i-1}{2} = \binom{n}{3}$. The sum is a telescoping sum since: $\binom{n}{3} - \binom{n-1}{3} = \binom{n-1}{2}.$