Fall AMC 10 B, 2021

Because $\angle RBC$ is complementary to both $\angle RCB$ and $\angle PBA$, those two angles are congruent. Therefore $\triangle BAP \cong \triangle CBQ$ by ASA, so $CQ = BP = 13$. Let $d = CR$; then $QR = 13 - d$, so the Altitude-to-Hypotenuse Theorem yields $6^2 = d(13 - d)$, which has solutions $d = 4$ and $d = 9$. Because $QR < RC$, in fact $d = 9$. It follows that the area of the square is
$$BC^2 = BR^2 + RC^2 = 6^2 + 9^2 = 117.$$

Let $c = BQ$ and $s = AB$. Because $\triangle BRQ$ is similar to $\triangle BAP$, it follows that $\frac{6}{c} = \frac{s}{13}$, so $cs = 78$. As before, $CQ = BP = 13$, so by the Pythagorean Theorem, $s^2 + c^2 = 169$. Then $(s+c)^2 = 169 + 2 \cdot 78 = 325$, so $s + c = 5\sqrt{13}$. Similarly $(s-c)^2 = 169 - 2 \cdot 78 = 13$, so $s - c = \sqrt{13}$. Solving this system of equations yields $s = 3\sqrt{13}$, and the area of the square is $s^2 = 117$.