It is obvious that a and b must be non-negative.
Suppose that 3a+7b=n2. We can assume that n is positive. We first work modulo 4. Since 3a+7b=n2, it follows that
n2≡(−1)a+(−1)b(mod4)
Since no square can be congruent to 2 modulo 4, it follows that we have either (i) a is odd and b is even or (ii) a is even and b is odd.
Case (i): Let b=2c. Then
3a=(n−7c)(n+7c).
It cannot be the case that 3 divides both n−7c and n+7c. But each of these is a power of 3. It follows that n−7c=1, and therefore
3a=2⋅7c+1.
If c=0, then a=1, and we obtain the solution a=1,b=0. So suppose that c≥1. Then 3a≡1(mod7). This is impossible, since the smallest positive value of a such that 3a≡1(mod7) is given by a=6, and therefore all a such that 3a≡1(mod7) are even, contradicting the fact that a is odd.
Case (ii): Let a=2c. Then
7b=(n−3c)(n+3c).
Thus each of n−3c and n+3c is a power of 7. Since 7 cannot divide both of these, it follows that n−3c=1, and therefore
7b=2⋅3c+1.
Look first at the case c=1. Then b=1, and we obtain the solution a=2,b=1. So from now on we may assume that c>1. Then 7b≡1(mod9). The smallest positive integer b such that 7b≡1(mod9) is given by b=3. It follows that b must be a multiple of 3. Let b=3d. Note that d is odd, so in particular d≥1.
Let y=7d. Then y3−1=2⋅3c, and therefore
2⋅3c=(y−1)(y2+y+1).
It follows that y−1=2⋅3u for some positive u, and that y2+y+1=3v for some v≥2. But since
3y=(y2+y+1)−(y−1)2,
it follows that 3∣y, which is impossible since 3∣(y−1).