Romanian Mathematical Olympiad - District Round

Assume $AB = BA$. We obtain $(A+B)^2 = 2AB$ and $(A+B)^3 = 2AB(A+B) = 2A^2B + 2AB^2 = O_3$. Then, $(A+B)(A-B) = A^2 - B^2 = O_3$. From Sylvester rank inequality, we get $\text{rank}(A+B) + \text{rank}(A-B) \le 3$. Thus, $\text{rank}(A+B) \le 1$ or $\text{rank}(A-B) \le 1$.

Suppose $\text{rank}(A+B) \le 1$. If $\text{rank}(A+B) = 0$, that is $A+B = O_3$, then $AB = -A^2 = O_3$. If $\text{rank}(A+B) = 1$, then there are two nonzero matrices, $C \in \mathcal{M}_{3,1}(\mathbb{C})$ and $D \in \mathcal{M}_{1,3}(\mathbb{C})$, such that $A+B = CD$. It follows $(A+B)^2 = (CD)(CD) = C(DC)D = \text{Tr}(A+B)(A+B)$. We obtain $O_3 = (A+B)^3 = \text{Tr}(A+B)(A+B)^2$. If $\text{Tr}(A+B) \ne 0$ then $(A+B)^2 = O_3$. If $\text{Tr}(A+B) = 0$ then, from the relation $(A+B)^2 = \text{Tr}(A+B)(A+B)$, we also obtain $(A+B)^2 = O_3$. Therefore $AB = O_3$.

The case where $\text{rank}(A-B) \le 1$ can be analogously treated.

A counterexample for the converse implication:
$$A = \begin{pmatrix} 0 & 0 & 0 \\ 0 & 0 & 1 \\ 0 & 0 & 0 \end{pmatrix} \quad \text{and} \quad B = \begin{pmatrix} 0 & 1 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{pmatrix}.$$
We have $A^2 = B^2 = AB = O_3$, but $AB \ne BA$.