5. A.

It is known that $\sin x \neq 0$, so we have $a \sin ^{2} x + b \cos x + c = 0$, which can be rewritten as $a \cos ^{2} x - b \cos x - (a + c) = 0$.
Given that $\alpha, \beta$ are the two roots, we know that $\cos \alpha, \cos \beta$ are the two distinct roots of the equation
$$a x^{2} - b x - (a + c) = 0$$

( $|\alpha \pm \beta| \neq k \pi$ ).
Thus, by Vieta's formulas, we have
$$\cos \alpha + \cos \beta = \frac{b}{a} \cdot \cos \alpha \cdot \cos \beta = -\frac{a + c}{a} .$$

Therefore, $\cos ^{2} \alpha + \cos ^{2} \beta = 2 + \frac{b^{2} + 2 a c}{a^{2}}$.