(a) Since $a+b^{2}$ is an integer, $-\frac{1}{b}+\frac{3 b}{a}$ is also an integer. This can be written as $\frac{-a+3 b^{2}}{a b}$. We see that $a b$ is a divisor of $3 b^{2}-a$. In particular, $b$ is a divisor of $3 b^{2}-a$, and thus $b \mid a$. But that means $b^{2}$ is a divisor of $a b$ and therefore also of $3 b^{2}-a$, which implies $b^{2} \mid a$. We can now write $a=m b^{2}$ with $m$ a positive integer. Then $m b^{3}$ is a divisor of $3 b^{2}-m b^{2}$, so $m b$ is a divisor of $3-m$. This implies that $m$ is a divisor of 3 (i.e., $m=1$ or $m=3$) and that $b$ is a divisor of $3-m$.
First, assume $m=3$. Then $a=3 b^{2}$. Substituting this gives $M\left(3 b^{2}, b\right)=3 b^{2}-\frac{1}{b}+b^{2}+\frac{1}{b}=4 b^{2}$, which is the square of $2 b$.
Now assume $m=1$. From $b \mid 3-m$ we get $b=1$ or $b=2$. In the first case, $a=1$ and in the second case, $a=4$. Substituting the first possibility gives $M(1,1)=1-1+1+3=4$, which is a square. Substituting the second possibility gives $M(4,2)=4-\frac{1}{2}+4+\frac{3}{2}=9$, which is also a square.
We conclude that $M(a, b)$ is a square in all cases.
(b) Take $a=4$ and $b=-2$. Then $M(4,-2)=7$. This is a positive integer, but not a square. After all the work in part (a), this answer is not hard to find. You know that $a$ must be of the form $m b^{2}$, now with $m \in \mathbb{Z}$, and that $m$ must be a divisor of 3. Moreover, $m=3$ does not work, as it always results in a square. The rest of the possibilities for $m$ can be simply tested.