HMMT February

Let $x=A P$ and $y=B P$. Rotate $\triangle B A P$ by $90^{\circ}$ around $B$ to get $\triangle B C Q$. Then, $\triangle B P Q$ is rightisosceles, and from $\angle B Q C=135^{\circ}$, we get $\angle P Q C=90^{\circ}$. Therefore, by Pythagorean's theorem, $P C^{2}=x^{2}+2y^{2}$. Similarly, $P D^{2}=y^{2}+2x^{2}$. Thus, $y^{2}=\frac{2P C^{2}-P D^{2}}{3}=21$, and similarly $x^{2}=102 \Longrightarrow xy=3\sqrt{238}$. Thus, by the Law of Cosines, the area of the square is $$\begin{aligned} A B^{2} & =A P^{2}+B P^{2}-2 \cos \left(135^{\circ}\right)(A P)(B P) \\ & =x^{2}+y^{2}+\sqrt{2}xy \\ & =123+6\sqrt{119} \end{aligned}$$