Solution:
Answer: $\frac{1}{4}$

After dividing the equation by $x^{2}$, we can rearrange it as
$$\left(x+\frac{1}{x}\right)^{2}+a\left(x+\frac{1}{x}\right)-b-2=0$$
Let $y = x + \frac{1}{x}$. We can check that the range of $x + \frac{1}{x}$ as $x$ varies over the nonzero reals is $(-\infty, -2] \cup [2, \infty)$. Thus, the following equation needs to have a real root:
$$y^{2} + a y - b - 2 = 0.$$
Its discriminant, $a^{2} + 4(b + 2)$, is always positive since $a, b \geq 0$. Then, the maximum absolute value of the two roots is
$$\frac{a + \sqrt{a^{2} + 4(b + 2)}}{2}.$$
We need this value to be at least $2$. This is equivalent to
$$\sqrt{a^{2} + 4(b + 2)} \geq 4 - a.$$
We can square both sides and simplify to obtain
$$2a \geq 2 - b$$
This equation defines the region inside $[0, 1] \times [0, 1]$ that is occupied by $S$, from which we deduce that the desired area is $1/4$.