Let us solve the given system of equations for numbers $a, b, c$. From the first two equations we obtain: $a - b = B^2 - A^2$. Let us add this to the third equation and find out, that
$$a = \frac{1}{2}(B^2 + C^2 - A^2).$$
Similarly, or thinking symmetrically, we can find out that
$$b = \frac{1}{2}(C^2 + A^2 - B^2) \text{ and } c = \frac{1}{2}(A^2 + B^2 - C^2).$$
The smallest possible sum $A^2 + B^2 + C^2$ can be received for three smallest various squares of integers:
$A^2 = 0, B^2 = 1, C^2 = 4$. But for these three squares, numbers $a, b, c$ will not be integers, for example:
$a = \frac{1}{2}(1 + 4 - 0) = \frac{5}{2}$. The following three squares with the next bigger sum are the numbers:
$A^2 = 0, B^2 = 1, C^2 = 9$. For this triple, the values of $a, b, c$ are integers, hence desired:
$$a = \frac{1}{2}(1 + 9 - 0) = 5, \quad b = \frac{1}{2}(9 + 0 - 1) = 4, \quad c = \frac{1}{2}(0 + 1 - 9) = -4.$$
Answer: $a = 5,\ b = 4,\ c = -4$.