Romanian Mathematical Olympiad

a) From $(A - B)^2 = O_2$ follows $\det(A - B) = 0$ and $\mathrm{Tr}(A - B) = 0$, hence $\mathrm{Tr}(A) = \mathrm{Tr}(B) =: a$.
Denote $b = \det(A) - \det(B)$. Then
$$\begin{cases} A^2 - aA + \det(A)I_2 = O_2 \\ B^2 - aB + \det(B)I_2 = O_2 \end{cases}$$
This shows that $\det(A^2 - B^2) = \det(a(A - B) - bI_2)$. On the other hand, $\det(a(A - B) - bI_2) = a^2 \det(A - B) - ab \mathrm{Tr}(A - B) + b^2 = b^2$, whence $\det(A^2 - B^2) = (\det(A) - \det(B))^2$.

b) Let $f : \mathbb{R} \to \mathbb{R}$ be the function given by
$$f(x) = \det(A^2 - B^2 + x(AB - BA)), \quad x \in \mathbb{R}.$$
Then $f(x) = \det(A^2 - B^2) + cx + \det(AB - BA)x^2$, $x \in \mathbb{R}$, where $c$ is a real constant. From $f(1) = f(-1) = \det(A - B)\det(A + B) = 0$ follows $c = 0$ and $\det(A^2 - B^2) + \det(AB - BA) = 0$. Now (a) leads to $(\det(A) - \det(B))^2 = -\det(AB - BA)$, whence the conclusion.