Solution:
We write the equation in the form
a2+2ap+p2+b(3a+2p)=0
Hence
b=3a+2p−(a+p)2
is an integer. This shows that 3a+2p divides (a+p)2 and hence also divides (3a+3p)2. But, we have
(3a+3p)2=(3a+2p+p)2=(3a+2p)2+2p(3a+2p)+p2
It follows that 3a+2p divides p2. Since p is a prime, the only divisors of p2 are ±1,±p and ±p2. Since p>3, we also have p=3k+1 or 3k+2.
Case 1: Suppose p=3k+1. Obviously 3a+2p=1 is not possible. In fact, we get 1=3a+2p=3a+2(3k+1)⇒3a+6k=−1 which is impossible. On the other hand 3a+2p=−1 gives 3a=−2p−1=−6k−3⇒a=−2k−1 and a+p=−2k−1+3k+1=k.
Thus b=3a+2p−(a+p)2=k2. Thus (a,b)=(−2k−1,k2) when p=3k+1. Similarly, 3a+2p=p⇒3a=−p which is not possible. Considering 3a+2p=−p, we get 3a=−3p or a=−p⇒b=0. Hence (a,b)=(−3k−1,0) where p=3k+1.
Let us consider 3a+2p=p2. Hence 3a=p2−2p=p(p−2) and neither p nor p−2 is divisible by 3. If 3a+2p=−p2, then 3a=−p(p+2)⇒a=−(3k+1)(k+1).
Hence a+p=(3k+1)(−k−1+1)=−(3k+1)k. This gives b=k2. Again (a,b)=(−(k+1)(3k+1),k2) when p=3k+1.
Case 2: Suppose p=3k−1. If 3a+2p=1, then 3a=−6k+3 or a=−2k+1. We also get
b=1−(a+p)2=1−(−2k+1+3k−1)2=−k2
and we get the solution (a,b)=(−2k+1,k2). On the other hand 3a+2p=−1 does not have any integral solution for a. Similarly, there is no solution in the case 3a+2p=p. Taking 3a+2p=−p, we get a=−p and hence b=0. We get the solution (a,b)=(−3k+1,0). If 3a+2p=p2, then 3a=p(p−2)=(3k−1)(3k−3) giving a=(3k−1)(k−1) and hence a+p=(3k−1)(1+k−1)=k(3k−1). This gives b=−k2 and hence (a,b)=(3k−1,−k2). Finally 3a+2p=−p2 does not have any solution.