Since $f(x)$ is an odd function defined on the real number set $\mathbb{R}$:
- For $x > 0$, we have $f(x) = x^2 - 2x - 3$. The axis of symmetry for the parabola defined by this function is at $x = 1$, and the parabola opens upwards. Setting $x^2 - 2x - 3 = 0$ yields solutions $x = 3$ and $x = -1$. Since $x > 0$, we discard the solution $x = -1$.
- When $x \leq 0$, the function's graph will be mirrored about the y-axis (due to it being an odd function), with an axis of symmetry at $x = -1$ and the parabola opening downwards. Solving $f(x) = 0$ for $x \leq 0$ gives us $x = -3$ and $x = 1$. Again, we discard the solution $x =1$ because $x \leq 0$. By the definition of an odd function, at $x = 0$, $f(x) = 0$.

Thus, when $x \leq 0$, the set of solutions for the inequality $f(x)\geq0$ is $[-3, 0]$.

Now, to find the number of integer solutions within the interval $[-3, 0]$, we list them as: $-3, -2, -1, 0$. There are $4$ such integers.

Hence, the final answer is $\boxed{4}$, which corresponds to option A.