Solution:
We have the trivial solutions (a,b,c)=(0,0,0) and (a,b,c)=(1,1,1), as well as the solution (a,b,c)=(127,106,43) and its cyclic permutations.
The case a=b=c=0 works. Without loss of generality, a=max{a,b,c}. If b and c are both zero, it's obvious that we have no solution. So, via the inequality
a2<a2+2b+c<(a+2)2
we find that a2+2b+c=(a+1)2⟹2a+1=2b+c. So,
a=b+2c−1.
Let c=2k+1 with k≥0; plugging into the given, we find that
b2+b+2+5kand4k2+6k+3b+1
are both perfect squares. Multiplying both these quantities by 4, and setting x=2b+1 and y=4k+3, we find that
x2+5y−8 and y2+6x−11
are both even squares.
We may assume x,y≥3. We now have two cases, both of which aren't too bad:
- If x≥y, then x2<x2+5y−8<(x+3)2. Since the square is even, x2+5y−8=(x+1)2. Then, x=25y−9 and we find that y2+15y−38 is an even square. Since y2<y2+15y−38<(y+8)2, there are finitely many cases to check. The solutions are (x,y)=(3,3) and (x,y)=(213,87).
- Similarly, if x≤y, then y2<y2+6x−11<(y+3)2, so y2+6x−11=(y+1)2. Then, y=3x−6 and we find that x2+15x−38 is a perfect square. Amusingly, this is the exact same thing (whether this is just a coincidence due to me selecting the equality case to be x=y, I'm not sure). Here, the solutions are (x,y)=(3,3) and (x,y)=(87,255).
Converting back, we see the solutions are (0,0,0), (1,1,1) and (127,106,43) and its cyclic permutations.