Solution:
Solution I. Adding $-3S+24$ to both sides makes the inequality equivalent to
$$(a-2)^2 + (b-2)^2 + (c-2)^2 + (d-2)^2 + (e-2)^2 + (f-2)^2 \leq (S-12)^2.$$
Substituting $A = a-2$, $B = b-2$, etc., this is the same as
$$A^2 + B^2 + C^2 + D^2 + E^2 + F^2 \leq (A+B+C+D+E+F)^2.$$
On the left side we have the sum of squares of nonnegative numbers and on the right side we have the square of the sum. The latter is always larger except when all pairwise products $AB, AC, DF, \ldots$ are zeros. This happens when all but one of $A, B, C, D, E, F$ are zero, correspondingly, when all but one of $a, b, c, d, e, f$ are $2$.

Solution II. The given inequality can be written as
$$a^2 + b^2 + c^2 + d^2 + e^2 + f^2 \leq (S-10)^2 + 20.$$
Now, for any two numbers $x, y \geq z$, we have
$$x^2 + y^2 \leq z^2 + (x+y-z)^2.$$
Indeed, this is equivalent to
$$x^2 - z^2 \leq (x+y-z)^2 - y^2 \Leftrightarrow (x-z)(x+z) \leq (x-z)(x+2y-z) \Leftrightarrow 0 \leq 2(x-z)(y-z),$$
and the last is true because $x \geq z$ and $y \geq z$. Note that (1) replaces the numbers $(x, y)$ by $(z, x+y-z)$ without changing the sum of the two numbers, but increases the sum of their squares. In the original problem, we do this for $a, b \geq 2$: we replace $(a, b)$ by $(2, a+b-2)$:
$$a^2 + b^2 \leq 2^2 + (a+b-2)^2.$$
We then do the same for $(a+b-2, c)$: replace them by $(2, a+b+c-4)$, and so on. In the end, we will have replaced five of the original numbers by $2$'s, and the last by $a+b+c+d+e+f-10 = S-10$.
$$\Rightarrow a^2 + b^2 + c^2 + d^2 + e^2 + f^2 \leq 2^2 + 2^2 + 2^2 + 2^2 + 2^2 + (S-10)^2 = 20 + (S-10)^2.$$
Equality is achieved if and only if there are equalities each time we apply (1), i.e., five of the given numbers are $2$'s, and the remaining number is therefore $S-10$.

Solution III. We first show the following inequality:
$$x(x-1) + y(y-1) \leq (x+y-2)(x+y-3) + 2, \quad x, y \geq 2$$
Consider two complete graphs, with $x$ and $y$ vertices, respectively. (A complete graph has all possible edges drawn.) Thus we have $\binom{x}{2}$ and $\binom{y}{2}$ edges in the two graphs. If we glue the graphs together on an edge, we produce a new graph with $x+y-2$ vertices. Count edges: the original configuration had $\binom{x}{2} + \binom{y}{2}$ edges, while the new configuration has at most $\binom{x+y-2}{2}$ edges. Since we lost an edge when we glued the two graphs together, we conclude that
$$\binom{x}{2} + \binom{y}{2} \leq \binom{x+y-2}{2} + 1$$
This is equivalent to (2) after multiplying by $2$. Equality is attained if and only if the new graph is also complete, i.e., one of the original graphs must have been just an edge ($x=2$ or $y=2$). From here, apply consecutively (2) to the desired inequality. Again, maximum is attained if and only if five of the given numbers are $2$'s.