11. Let the three sides of $\triangle ABC$ be $a, b, c$, then it is easy to prove
$$\begin{array}{l} \tan \frac{A}{2}=\frac{a^{2}-(b-c)^{2}}{4 S} \\ \tan \frac{B}{2}=\frac{b^{2}-(c-a)^{2}}{4 S} \\ \tan \frac{C}{2}=\frac{c^{2}-(a-b)^{2}}{4 S} \end{array}$$

Thus, the original inequality becomes
$$a^{2}-(b-c)^{2}+b^{2}-(c-a)^{2}+c^{2}-(a-b)^{2} \leqslant 18 R r$$

Make the substitution $a=y+z, b=z+x, c=x+y$. Where $x, y, z$ are positive numbers.
Since $S=\frac{1}{2}(a+b+c) r=\frac{a b c}{4 R}$, we have $R r=\frac{a b c}{2(a+b+c)}=$ $\frac{(y+z)(z+x)(x+y)}{4(x+y+z)}$. Thus, inequality (1) becomes
$$\begin{array}{l} 4(x y+y z+z x) \leqslant \frac{9(y+z)(z+x)(x+y)}{4(x+y+z)} \Leftrightarrow \\ 8(x y+y z+z x)(x+y+z) \leqslant 9(y+z)(z+x)(x+y) \Leftrightarrow \\ 6 x y z \leqslant y\left(z^{2}+x^{2}\right)+z\left(x^{2}+y^{2}\right)+x\left(y^{2}+z^{2}\right) \end{array}$$

Since $z^{2}+x^{2} \geqslant 2 z x, x^{2}+y^{2} \geqslant 2 x y, y^{2}+z^{2} \geqslant 2 y z$, we have $6 x y z \leqslant y\left(z^{2}+x^{2}\right)+$ $z\left(x^{2}+y^{2}\right)+x\left(y^{2}+z^{2}\right)$, thus the original inequality holds.