Solution:

a) If $\triangle ABC$ is acute, then by the Law of cosines for $\triangle AHB$ we get that
$$AB^2 = AH^2 + BH^2 - 2 AH \cdot BH \cos(\pi - \gamma)$$
Since $AB = 2R \sin \gamma$ and $CH = 2R \cos \gamma$ (by the Extended Law of sines), we obtain $AB^2 + CH^2 = 4R^2$. Therefore
$$4R^2 = AH^2 + BH^2 + CH^2 + \frac{AH \cdot BH \cdot CH}{R}$$
Then $4R^3 = 7R + 3$, i.e. $(R+1)(2R+1)(2R-3) = 0$, whence $R = \frac{3}{2}$.
If $\triangle ABC$ is obtuse, then we get analogously that
$$4R^2 = AH^2 + BH^2 + CH^2 - \frac{AH \cdot BH \cdot CH}{R}$$
and therefore $4R^3 = 7R - 3$, i.e. $(R-1)(2R-1)(2R+3) = 0$. Since $3 = AH \cdot BH \cdot CH < (2R)^3$, we conclude that $R = 1$.
The existence of $\triangle ABC$ with $R = \frac{3}{2}$ and $R = 1$ follows from b).

b) Denote by $S$ the area of $\triangle ABC$. Since $S = \frac{AB \cdot BC \cdot CA}{4R}$, we have
$$S^2 = \frac{(4R^2 - AH^2)(4R^2 - BH^2)(4R^2 - CH^2)}{16R^2}$$
Setting $x = AH^2$, $y = BH^2$, $z = CH^2$ and $t = 4R^2$, we get
$$S^2 = \frac{t^3 - 7t^2 + t(xy + yz + zx) - 9}{4t}$$
Without loss of generality we may assume that $x \geq y \geq z$. Then $x \geq \frac{7}{3}$ and therefore
$$xy + yz + zx = \frac{9}{x} + x(7 - x) = 15 - \frac{(x-3)^2(x-1)}{x} \leq 15$$
where the equality is attained if $x = 3$. Hence
$$S^2 \leq \frac{t^3 - 7t^2 + 15t - 9}{4t}$$
Since $R = \frac{3}{2}$ or $R = 1$, we conclude that $S_{\max} = \sqrt{8}$ and it is achieved for an acute $\triangle ABC$ with $R = \frac{3}{2}$, $AH = BH = \sqrt{3}$ and $CH = 1$. The sides of this triangle are $\sqrt{6}$, $\sqrt{6}$ and $\sqrt{8}$.