a. For $a = 0$, we have
$$\begin{cases} x = yz, \\ y = zx, \\ z = xy. \end{cases}$$
If one of three numbers is equal to $0$ then the other numbers are equal to $0$ too. We consider the case $xyz \neq 0$ and multiply the equations, side by side, we get $xyz = 1$ then
$$x^2 = y^2 = z^2 = 1.$$
From these identities, we conclude that the solutions of the system are
$$(0, 0, 0), (1, 1, 1), (-1, -1, 1)$$
and permutations. We can easily check that there are 5 different solutions for this system.

b. Clearly, $x = y = z = 0$ satisfies the system. For $a > 1$, first of all it is easy to see that if any number equals $0$, the other numbers also equal $0$. Now, we try to transform this system to an equation for $z$.
$$\begin{cases} x = ay + yz, \\ y - az = z(ay + yz), \\ z - a(ay + yz) = y(ay + yz), \end{cases} \Leftrightarrow \begin{cases} x = ay + yz, \\ y = \frac{az}{1 - az - z^2}, \\ z - a(ay + yz) = y(ay + yz). \end{cases}$$
Hence,
$$z - (a^2 + az) \frac{az}{1 - az - z^2} = (a + z) \left( \frac{az}{1 - az - z^2} \right)^2$$
or
$$z^4 + (a^2 + 2a)z^3 + 2(a^3 - 1)z^2 + (a^4 - a^3 - a^2 - 2a)z + (1 - a^3) = 0.$$
Note that the original system has two roots $x = y = z = 0$ and $x = y = z = 1 - a$ so the above equation must have a root $z = 1 - a$. Hence, we have
$$z^3 + (a^2 + a + 1) z^2 + (a^3 - 1) z - a^2 - a - 1 = 0.$$
Consider the polynomial $f(z) = z^3 + (a^2 + a + 1) z^2 + (a^3 - 1) z - a^2 - a - 1$. By Rolle's theorem, we have
$$f(-\infty) < 0,\ f(-2a) > 0,\ f(0) < 0,\ f(+\infty) > 0$$
which means $f$ has three distinct real roots. Note that from the simplified system, it is clear that when we get $z$; $x$, $y$ are identified uniquely. Therefore, the system has exactly 5 different solutions when $a > 1$.
$\Box$