We can rearrange the equation into $4x^2=40\lfloor x \rfloor-51$. Obviously, the RHS is an integer, so $4x^2=n$ for some integer $n$. We can therefore make the substitution $x=\frac{\sqrt{n}}{2}$ to get
$$40\left\lfloor \frac{\sqrt{n}}{2}\right\rfloor-51=n$$
(We'll try the case where $x=-\frac{\sqrt{n}}{2}$ later.) Now let $a\le\frac{\sqrt{n}}{2}0\]
The first inequality simplifies to $(2a-10)^2\le 49$, or $|2a-10|\le 7$. Since $2a-10$ is even, we must have $|2a-10|\le 6$, so $|a-5|\le 3$. Therefore, $2\le a\le 8$. The second inequality simplifies to $(2a-8)^2 > 9$, or $|2a-8| > 3$. Therefore, as $2a-8$ is even, we have $|2a-8|\ge 4$, or $|a-4|\ge 2$. Hence $a\ge 6$ or $a\le 2$. Since both inequalities must be satisfied, we see that only $a=2$, $a=6$, $a=7$, and $a=8$ satisfy both inequalities. Each will lead to a distinct solution for $n$, and thus for $x$, for a total of $4$ positive solutions.
Now let $x=-\frac{\sqrt{n}}{2}$. We have
$$40\left\lfloor -\frac{\sqrt{n}}{2}\right\rfloor-51=n$$
Since $\lfloor -x\rfloor=-\lceil x\rceil$, this can be rewritten as
$$-40\left\lceil \frac{\sqrt{n}}{2}\right\rceil-51=n$$
Since $n$ is positive, the least possible value of $\left\lceil \frac{\sqrt{n}}{2}\right\rceil$ is $1$, hence $-40\left\lceil \frac{\sqrt{n}}{2}\right\rceil-51\le -91$, i.e., it must be negative. But it is also equal to $n$, which is positive, and this is a contradiction. Therefore, there are no negative roots.
The total number of roots to this equation is thus $4$, or $\boxed{(\text{E})}$.