Solution:

Clearly, the expression is not divisible by $5$ when $p = 5$. For any prime $p$ other than $5$, note that $p^{4} \equiv 1 \pmod{5}$, so $p^{4} + 5p^{3} + 4 \equiv 0 \pmod{5}$. Hence, the problem reduces to counting the number of primes less than $50$ other than $5$. There are $13$ such primes.