a.
The answer is $r = \frac{abc^2}{4R(a+b)^2}$ or any equivalent expression.

Let $O$, $O_1$, $O_2$ be the centres of $K$, $K_1$, $K_2$ respectively. Let $R_1$ and $R_2$ be the radii of $K_1$ and $K_2$ respectively. We have
$$R_1 = AO_1 = \frac{AC}{2 \cos \angle CAO_1} = \frac{b}{2 \sin A} = \frac{bR}{a}$$
by the extended sine law. By symmetry, we have $R_2 = \frac{aR}{b}$.

Now, let $T$ be the tangential point of $K$ with $AB$. By Pythagorus' theorem, we have
$$AT = \sqrt{OO_1^2 - (AO_1 - TO)^2} = \sqrt{(R_1 + r)^2 - (R_1 - r)^2} = 2\sqrt{R_1r}.$$
Similarly, we have $BT = 2\sqrt{R_2r}$. This yields
$$c = AB = 2\sqrt{R_1r} + 2\sqrt{R_2r},$$
$$r = \frac{c^2}{4(\sqrt{R_1} + \sqrt{R_2})^2} = \frac{c^2}{4(R_1 + R_2 + 2\sqrt{R_1 R_2})} = \frac{abc^2}{4R(a+b)^2}.$$

b.
We have $\angle C = 90^\circ$.

Note that $\frac{ab}{(a+b)^2} \le \frac{1}{4}$ (since this is the same as $(a-b)^2 \ge 0$) and $c \le 2R$.

By part (a), we have
$$r = \frac{abc^2}{4R(a+b)^2} \le \frac{4R^2}{16R} = \frac{R}{4}.$$
Therefore, equality should hold. This means $a = b$ and $c = 2R$. In order that $c$ is the diameter of the circumcircle, we must have $\angle C = 90^\circ$.