a) Assume A=(aij)1≤i,j≤n and A2=(mij)1≤i,j≤n. The property AT=−A leads to the relations aji=−aij, for i,j=1,…,n, that is A is antisymmetric. Then
mii=j=1∑naijaji=−j=1∑naij2,i=1,…,n.
If A2=On, then mii=0, i=1,…,n. Since A∈Mn(R), we obtain aij=0, for i,j=1,…,n. So A=On.
b) From the assumption, A=B∗, where B∗ is the adjoint of B. Since n is an odd integer number, we obtain det(A)=det(AT)=det(−A)=(−1)ndet(A)=−det(A). Then we obtain det(A)=0. Therefore det(BB∗)=det(B)⋅det(B∗)=det(B)⋅det(A)=0. From the relation BB∗=det(B)In, we get det(B)=0. Hence \rank(B)≤n−1 and BB∗=On.
Case 1. If \rank(B)≤n−2, then B∗=On. Thus, A2=(B∗)2=On.
Case 2. If \rank(B)=n−1, then B∗=On. From the Sylvester rank inequality, we have \rank(B∗)≤\rank(BB∗)+n−\rank(B)=1. We conclude \rank(B∗)=1 and (B∗)2=\tr(B∗)B∗. But \tr(B∗)=\tr(A)=0 because aii=−aii, i=1,2,…,n. Finally, we get A2=(B∗)2=On.