Problem 1778

a) Assume $A = (a_{ij})_{1 \le i,j \le n}$ and $A^2 = (m_{ij})_{1 \le i,j \le n}$. The property $A^T = -A$ leads to the relations $a_{ji} = -a_{ij}$, for $i, j = 1, \dots, n$, that is $A$ is antisymmetric. Then
$$m_{ii} = \sum_{j=1}^{n} a_{ij} a_{ji} = - \sum_{j=1}^{n} a_{ij}^2, \quad i = 1, \dots, n.$$
If $A^2 = O_n$, then $m_{ii} = 0$, $i = 1, \dots, n$. Since $A \in \mathcal{M}_n(\mathbb{R})$, we obtain $a_{ij} = 0$, for $i, j = 1, \dots, n$. So $A = O_n$.

b) From the assumption, $A = B^*$, where $B^*$ is the adjoint of $B$. Since $n$ is an odd integer number, we obtain $\det(A) = \det(A^T) = \det(-A) = (-1)^n \det(A) = -\det(A)$. Then we obtain $\det(A) = 0$. Therefore $\det(BB^*) = \det(B) \cdot \det(B^*) = \det(B) \cdot \det(A) = 0$. From the relation $BB^* = \det(B)I_n$, we get $\det(B) = 0$. Hence $\text{(B) n-1}$ and $BB^* = O_n$.

Case 1. If $\text{(B) n-2}$, then $B^* = O_n$. Thus, $A^2 = (B^*)^2 = O_n$.

Case 2. If $\text{(B) = n-1}$, then $B^* \ne O_n$. From the Sylvester rank inequality, we have $\text{(B *) (BB *) + n - (B) = 1}$. We conclude $\text{(B *) = 1}$ and $\text{(B *) 2 = (B *)B *}$. But $\text{(B *) = (A) = 0}$ because $a_{ii} = -a_{ii}$, $i = 1, 2, \dots, n$. Finally, we get $A^2 = (B^*)^2 = O_n$.