a) Define C=2A−In and D=2B−In. We have CD−In=−(DC−In)=2(AB−BA). As n is odd, det(CD−In)=−det(DC−In).
As for any two square matrices X,Y, we have det(XY−In)=det(YX−In), in our case we get det(CD−In)=−det(CD−In), that is det(CD−In)=0. Consequently det(AB−BA)=0.
b) Define E=AB−BA. Suppose det(E)=0. We have AE+EA=A2B−BA2=A(A+B−BA)−(A+B−AB)A=AB−BA=E. As E is invertible, E−1AE+A=In. Using the property tr(E−1AE)=tr(A), we get tr(A)=2n. By symmetry, tr(B)=2n, implying tr(A)=tr(B), in contradiction with the condition b). That is det(AB−BA)=0.