Romanian Mathematical Olympiad

If $b = 0$, then $BA = aI_n$, so $A$ and $B$ are invertible. We will suppose $b \neq 0$.

Denote by $\sigma(X)$ the set of eigenvalues of a matrix $X \in \mathcal{M}_n(\mathbb{C})$. Let $\lambda \in \sigma(AB)$. Then
$$\det(BA - (b\lambda + a)I_n) = \det(bAB - b\lambda I_n) = b^n \det(AB - \lambda I_n) = 0,$$
implying $b\lambda + a \in \sigma(BA)$. As $\sigma(AB) = \sigma(BA)$, if $\lambda \in \sigma(AB)$, we get $b\lambda + a \in \sigma(AB)$.

Define $f : \mathbb{C} \to \mathbb{C}$, $f(z) = bz + a$, for all $z \in \mathbb{C}$, and for $k \in \mathbb{N}^*$, denote by $f^{[k]} = \underbrace{f \circ f \circ f \dots \circ f}_{k \text{ times}}$. We conclude inductively, that if $\lambda \in \sigma(AB)$, then $f^{[k]}(\lambda) \in \sigma(AB)$, for $k \in \mathbb{N}^*$.

Suppose $0 \in \sigma(AB)$. Then $f(0), f^{[2]}(0), \dots, f^{[n+1]}(0) \in \sigma(AB)$. As $\sigma(AB)$ has at most $n$ elements, there are $p, q \in \{1, 2, \dots, n+1\}$, $p < q$, such that $f^{[p]}(0) = f^{[q]}(0)$.

Because $f^{[k]}(z) = b^k z + a \frac{b^{k}-1}{b-1}$, $z \in \mathbb{C}$, $k \in \mathbb{N}^*$, we obtain $a \frac{b^p - 1}{b-1} = a \frac{b^q - 1}{b-1}$, that is $b^{q-p} = 1$, a contradiction.

In conclusion, $0 \notin \sigma(AB)$, so $\det(A) \cdot \det(B) = \det(AB) \neq 0$, and the result follows.