To prove $(ABC) = \frac{a^2}{2(\cot B + \cot C)}$, we recall that $\sin(B + C) = \sin(A)$ because $\angle A + \angle B + \angle C = 180^\circ$ and obtain
$$\begin{aligned} \cot B + \cot C &= \frac{\cos B \sin C + \cos C \sin B}{\sin B \sin C} = \frac{\sin(B + C)}{\sin B \sin C} \\ &= \frac{\sin A}{\sin B \sin C} = \frac{a}{b \sin C} \quad \text{(Sine Rule)} \\ &= \frac{a^2}{ab \sin C} = \frac{a^2}{2(ABC)}, \end{aligned}$$
the required result.

Here is an alternative way to prove this formula. Let $D$ be the foot of the altitude from $A$, $x = |AD|$, and $y = |CD|$ where $x$ is taken negative if $\angle B$ is obtuse and $y$ is taken negative if $\angle C$ is obtuse. We then have $a = x + y$, $\cot B = x/h$ and $\cot C = y/h$, hence $\cot B + \cot C = a/h$. Using $2(ABC) = ah$ this turns into the desired formula.

To show that $\cos A \cos B \cos C \le \frac{1}{8}$ for all acute-angled triangles $ABC$, we use the area formula shown above. Since $B, C$ are acute angles, $\cot B$ and $\cot C$ are positive and we can use the AM-GM inequality to obtain
$$\sqrt{\cot B \cot C} \le \frac{\cot B + \cot C}{2} = \frac{a^2}{4(ABC)},$$
with equality iff $\angle B = \angle C$. Whence
$$\begin{aligned} (ABC) &\le \frac{a^2}{4} \sqrt{\tan B \tan C}, \\ &= \frac{a}{2} \frac{\sqrt{\left(\frac{1}{2}ab \sin C\right) \left(\frac{1}{2}ac \sin B\right)}}{\sqrt{bc \cos B \cos C}} = \frac{a}{2} \frac{(ABC)}{\sqrt{bc \cos B \cos C}}, \end{aligned}$$
and so
$$\cos B \cos C \le \frac{a^2}{4bc}, \qquad (8)$$
with equality iff $\angle B = \angle C$. Similarly,
$$\cos C \cos A \le \frac{b^2}{4ca}, \quad \text{with equality iff } \angle C = \angle A \text{ and}$$
$$\cos A \cos B \le \frac{c^2}{4ab}, \quad \text{with equality iff } \angle A = \angle B.$$
Hence
$$(\cos A \cos B \cos C)^2 \le \frac{1}{64},$$
with equality iff $\angle A = \angle B = \angle C$, whence the desired inequality follows.

An alternative proof of inequality (8), not using the area formula we have shown in the first part, may use the Cosine Rule as follows:
$$\begin{aligned} \cos B \cos C &= \frac{(a^2 - b^2 + c^2)}{2ac} \cdot \frac{(a^2 + b^2 - c^2)}{2ab} \\ &= \frac{a^4 - (b^2 - c^2)^2}{4a^2bc} \le \frac{a^4}{4a^2bc} = \frac{a^2}{4bc}, \end{aligned}$$