Irish Mathematical Olympiad

Proceed as in the hint and let $P'$ denote the rotated position of $P$. Thus $|AP'| = 3$, $|BP'| = 5$ and $\angle PAP' = 90^\circ$. Since $PAP'$ is an isosceles right triangle with $|AP'| = |AP| = 3$, then $|PP'| = 3\sqrt{2}$.

The area of the triangle $PBP'$ is, by Heron's formula, given by
$$\sqrt{\left(\frac{3\sqrt{2}+12}{2}\right)\left(\frac{3\sqrt{2}-2}{2}\right)\left(\frac{3\sqrt{2}+2}{2}\right)\left(\frac{12-3\sqrt{2}}{2}\right)} = \frac{21}{2}$$
But the area of the triangle $PBP'$ is also given by $\frac{1}{2}|PB||PP'| \sin \angle BPP'$, which is equal to $\frac{21\sqrt{2}}{2} \sin \angle BPP'$ and so $\frac{1}{\sqrt{2}} = \sin \angle BPP'$. This gives $\angle BPP' = 45^\circ$, and so $\angle APB = 90^\circ$. From the right triangle $APB$ we get $|AB|^2 = 58$. Thus the area of the square is $58$.