a. First note that
$$2(a + b + c)^2 - 6(ab + bc + ca) = (a - b)^2 + (b - c)^2 + (c - a)^2 \geq 0$$
hence $ab + bc + ca \le \frac{1}{3}(a + b + c)^2 = 3$. Now use Cauchy-Schwarz with vectors $(\sqrt{a}, \sqrt{b}, \sqrt{c})$ and $(\sqrt{ab}, \sqrt{bc}, \sqrt{ca})$ to get
$$a\sqrt{b} + b\sqrt{c} + c\sqrt{a} \le \sqrt{a + b + c} \cdot \sqrt{ab + bc + ca} \le \sqrt{3} \cdot \sqrt{3} = 3.$$

b. Homogenising the inequality we want to prove that
$$\sqrt{a^3 b} + \sqrt{b^3 c} + \sqrt{c^3 a} + \sqrt{ab^3} + \sqrt{bc^3} + \sqrt{ca^3} \le \frac{2}{3}(a + b + c)^2.$$
We introduce $T(a, b) := a^2 + b^2 + 4ab - 3\sqrt{a^3 b} - 3\sqrt{ab^3}$, multiply the above by 3 and rearrange to obtain $T(a, b) + T(b, c) + T(c, a) \ge 0$. To show $T(a, b) \ge 0$, we define $x = \sqrt{a}$ and $y = \sqrt{b}$. This gives
$$T(a, b) = x^4 + y^4 + 4x^2 y^2 - 3x^3 y - 3x y^3 = (x - y)^2(x^2 - x y + y^2) \ge 0.$$
Similarly $T(b, c) \ge 0$ and $T(c, a) \ge 0$. This proves the result.