Romanian Mathematical Olympiad - Final Round

a) If $A$ is invertible or $A = O_n$ then clearly $AB - BA = O_n$. Assume $A \ne O_n$, with $\det(A) = 0$. Let $P \in \mathbb{C}[X]$ be the minimal polynomial of the matrix $A$. Since $P(0) = 0$ and $P \ne X$, the polynomial $P$ has the form $P = X^k + a_{k-1}X^{k-1} + \dots + a_1X$, where $2 \le k \le n$. From the relation $P(A)B = O_n$ and the hypothesis, we obtain
$$A^{k-1} + a_{k-1}A^{k-2} + \dots + a_2A + a_1AB = O_n. \quad (1)$$
Since $P$ is minimal, we have $a_1 \ne 0$ and $AB = -\frac{1}{a_1}(A^{k-1} + a_{k-1}A^{k-2} + \dots + a_2A)$. So $AB$ commutes with $A$. Therefore $A = A^2B = A(AB) = ABA$. Then, by multiplying with $BA$ the relation (1) to the right, we obtain
$$A^{k-1} + a_{k-1}A^{k-2} + \dots + a_2A + a_1AB^2A = O_n. \quad (2)$$
From (1) and (2) it results $a_1(AB^2A - AB) = O_n$. Since $a_1 \ne 0$, we get $AB^2A = AB$. Thus, $(AB - BA)^2 = (ABA)B - AB^2A - B(A^2B) + B(ABA) = AB - AB - BA + BA = O_n$.

Alternative solution.

a) From $\mathrm{rank}(A) = \mathrm{rank}(A^2) \le \mathrm{rank}(A^2) \le \mathrm{rank}(A)$, we obtain $\mathrm{rank}(A) = \mathrm{rank}(A^2)$. Denote $r = \mathrm{rank}(A)$. There are the matrices $X \in \mathcal{M}_{n,r}(\mathbb{C})$ and $Y \in \mathcal{M}_{r,n}(\mathbb{C})$, with $\mathrm{rank}(X) = \mathrm{rank}(Y) = r$, such that $A = XY$. We have $YX \in \mathcal{M}_r(\mathbb{C})$ and
$$r = \mathrm{rank}(A) = \mathrm{rank}(A^2) = \mathrm{rank}((XY)^2) = \mathrm{rank}(X(YX)Y) \le \mathrm{rank}(YX).$$
Hence $YX$ is an invertible matrix and from the relation $A^2B = A$ we find $YBX = I_r$. Therefore $ABA = (XY)B(XY) = X(YBX)Y = XY = A$. From the relations $A^2B = A$ and $ABA = A$, we get
$$\begin{aligned} (AB - BA)^2 &= (AB)^2 + (BA)^2 - AB^2A - BA^2B \\ &= (ABA)B + B(ABA) - AB^2A - B(A^2B) \\ &= AB + BA - AB^2A - BA = AB(I_n - BA) \end{aligned}$$
and
$$\begin{aligned} (AB - BA)^3 &= AB(I_n - BA)(AB - BA) \\ &= AB(AB - BA - BA^2B + (BA)^2) \\ &= AB(AB - BA - B(A^2B) + B(ABA)) \\ &= AB(AB - BA - BA + BA) = AB(AB - BA) \\ &= (ABA)B - AB^2A = AB - (AB)(BA) = AB(I_n - BA). \end{aligned}$$
Thus, $(AB - BA)^3 = (AB - BA)^2$. It follows that, if $\lambda \in \mathbb{C}$ is an eigenvalue of the matrix $AB - BA$, then $\lambda^3 = \lambda^2$, hence $\lambda \in \{0, 1\}$. But $\text{Tr}(AB - BA) = 0$. Then all the eigenvalues of the matrix $AB - BA$ are 0. Therefore $(AB - BA)^n = O_n$. From the proved relation $(AB - BA)^3 = (AB - BA)^2$ we deduce $(AB - BA)^2 = O_n$.

b) Let us define the matrices $A = \begin{pmatrix} O_{n-k} & O_{n-k,k} \\ O_{k,n-k} & I_k \end{pmatrix}$, $B = \begin{pmatrix} O_{n-k} & C \\ O_{k,n-k} & I_k \end{pmatrix}$, where $C \in \mathcal{M}_{n-k,k}(\mathbb{C})$ is an arbitrary matrix with $\mathrm{rank}(C) = k \le n-k$ (for $k=0$, we define $A = B = O_n$). We have $A^2B = AB = A$, $BA = B$ and $AB - BA = A - B = \begin{pmatrix} O_{n-k} & -C \\ O_{k,n-k} & O_k \end{pmatrix}$, so $\mathrm{rank}(AB - BA) = k$.