Solution:
$a_{n}$ satisfies a standard linear recurrence relation with general solution $a_{n}=n^{2}+A n+k$. But $a_{0}=0$, so $k=0$. Hence $a_{n}=n^{2}+A n$. If you are not familiar with the general solution, then you can guess this solution and prove it by induction.

Similarly, $b_{n}=B n+8$. Hence $a_{n}^{2}+b_{n}^{2}=n^{4}+2 A n^{3}+(A^{2}+B^{2}) n^{2}+16 B n+64$. If this is a square, then looking at the constant and $n^{3}$ terms, it must be $\left(n^{2}+A n+8\right)^{2}$. Comparing the other terms, $A=B= \pm 4$.