We start from the observation that for $k = 1, 2, \dots, p$ we have
$$\binom{p-1}{k-1} \equiv \pm 1 \pmod{p}. \qquad (1)$$
To see this, note that
$$p-1 \equiv -1 \pmod{p},$$
$$p-2 \equiv -2 \pmod{p},$$
$$\vdots \qquad \vdots$$
$$k \equiv -(p-k) \pmod{p},$$
hence, multiplying,
$$\frac{(p-1)!}{(k-1)!} \equiv \pm(p-k)! \pmod{p}$$
and (1) follows. This can be written in an equivalent form
$$\frac{k}{p} \binom{p}{k} \equiv \pm 1 \pmod{p},$$
which implies that
$$\binom{p}{k} = p \cdot \frac{a_k p \pm 1}{k},$$
for some integer $a_k$. Therefore, if $p$ satisfies the given conditions, we have
$$p \mid \sum_{k=1}^{p-1} \frac{(a_k p \pm 1)^2}{k^2},$$
or $p \mid \sum_{k=1}^{p-1} 1/k^2$, where by $1/m$ we understand the unique integer $1 \le l \le p-1$ satisfying $ml \equiv 1 \pmod{p}$. Now observe that $1/k^2 \equiv (1/k)^2 \pmod{p}$ and
$$\sum_{k=1}^{p-1} \frac{1}{k^2} \equiv \sum_{k=1}^{p-1} k^2 = \frac{p(p-1)(2p-1)}{6} \pmod{p},$$
which is divisible by $p$ if and only if $p \ge 5$.