Since the average of the sample $x, 1, y, 5$ is $2$,
we have $x+y=2$,
thus $xy \leq 1$,
therefore, the variance $S^2 = \frac{1}{4}[(x-2)^2+(y-2)^2+10]$
$= \frac{5}{2} + \frac{1}{4}(x^2+y^2) \geq \frac{5}{2} + \frac{1}{4} \cdot 2xy = \frac{5}{2} + \frac{1}{4} \times 2 = 3$,
equality holds if and only if $x=y=1$,
thus, the minimum value of the variance is $3$.
Hence, the answer is $\boxed{3}$.
To solve this, we find $x+y=2$, determine the minimum value of $xy$, and then calculate the minimum value of the variance accordingly.
This problem tests the calculation of the variance and average of data, as well as the application of basic inequalities, making it a foundational question.