For reasons of readability let us write $A_n = (n^2 + 11n - 4) \cdot n! + 33 \cdot 13^n + 4$. First, consider the value of $A_n$ modulo $8$ for $n \ge 4$. We have $8 \mid 1 \cdot 2 \cdot 3 \cdot 4 \cdot \dots \cdot n = n!$, so
$$A_n = (n^2 + 11n - 4) \cdot n! + 33 \cdot 13^n + 4 \equiv 0 + 1 \cdot 5^n + 4 \pmod{8}.$$
Since $5^2 \equiv 1 \pmod{8}$, we have $5^{2k} \equiv 1 \pmod{8}$ and $5^{2k-1} \equiv 5 \pmod{8}$ for all $k \in \mathbb{N}$. For even $n$ we get $A_n \equiv 5 \pmod{8}$ and for odd $n$ we have $A_n \equiv 1 \pmod{8}$. On the other hand a perfect square can only give the remainder of $0, 1$ or $4$ when divided by $8$, so all even $n \ge 4$ are out of consideration.

Now, consider the value of $A_n$ modulo $7$ for $n \ge 7$. We have $7 \mid n!$, so
$$A_n = (n^2 + 11n - 4) \cdot n! + 33 \cdot 13^n + 4 \equiv 0 + 5 \cdot (-1)^n + 4 \pmod{7}.$$
Since $(-1)^2 = 1 \pmod{7}$, we have $(-1)^{2k} = 1 \pmod{7}$ and $(-1)^{2k-1} = -1 \pmod{7}$ for all $k \in \mathbb{N}$. For even $n$ we get $A_n \equiv 2 \pmod{7}$, for odd $n$ we get $A_n \equiv 6 \pmod{7}$. A perfect square can only give the remainders of $0, 1, 2$ or $4$ when divided by $7$, so we can exclude all odd $n \ge 7$.

The remaining options are $n = 1, 2, 3$ and $5$.
If $n = 3$ we have
$$A_3 \equiv 3 \cdot 1 + 3 \cdot 3^3 + 4 \equiv 3 \pmod{5}.$$
A perfect square can only give the remainder of $0, 1$ or $4$ when divided by $5$, so $A_3$ is not a perfect square.
A similar reasoning helps us see that $n = 5$ does not work, since
$$A_5 \equiv (-4) \cdot 0 + 3 \cdot 3^5 + 4 \equiv 3 \pmod{5}.$$
If $n = 1$, then $A_1 = 441 = 21^2$. If $n = 2$, then $A_2 = 5625 = 75^2$. Thus, $n = 1$ and $n = 2$ are the only solutions.