a) From A3=B3 we get detA=detB=d and by Hamilton-Cayley theorem we obtain A2=aA−dI2, B2=bB−dI2, where a=trA, b=trB. As a consequence one can write A3=aA2−dA=(a2−d)A−adI2 and B3=(b2−d)B−bdI2, so (a2−d)A−adI2=(b2−d)B−bdI2 (*).
By right and left multiplication with B we get (a2−d)BA−adB=(b2−d)B2−bdB=(a2−d)AB−adB, so d=a2. In the same way, d=b2, and by (*), we deduce d=0 or a=b.
If d=0, then a=b=0, so A2=B2=O2, from where An=Bn, for any n≥2. So tr A=a=0=b=tr B and tr An=tr Bn, for all n≥2.
If d=0, then a=b and C2=aC−a2I2, C3=−a3I2, so C3k=(−a3)kI2, C3k−2=(−a3)k−1C, C3k−1=(−a3)k−1(aC−a2I2), ∀C∈{A,B},∀k≥1.
b) For A=(1011), B=(1101), we have AB=(2111), BA=(1112),
An=(10n1), Bn=(1n01) and tr An=tr Bn=2,∀n≥1.