(a) Let p(X,Y) be such a polynomial. There are two possible cases:
Case 1. If p(1,Y)=p(2,Y)=p(3,Y)=p(4,Y)=0. In this case
(X−1)(X−2)(X−3)(X−4)
is a factor of p(X,Y) and therefore degp(X,Y)≥4.
Case 2. If there exists a0∈{1,2,3,4} such that p(a0,Y)=0. Because p(a0,b)=0 for b∈{1,2,3,4}∖{a0}, the nonzero polynomial p(a0,Y) has 3 different roots and therefore degp(X,Y)≥degp(a0,Y)≥3.
Hence degp(X,Y)≥3.
To construct such a polynomial of degree 3, consider, as shown in the figure, the line of equation x+y−5=0 and the circle of equation (x−25)2+(y−25)2−25=0.

The graph obtained by union of this line and this circle passes by all the points of S. Therefore, the polynomial
p0(X,Y)=(X+Y−5)(X2+Y2−5X−5Y+10)
satisfies the condition p0(a,b)=0 for all (a,b)∈S and is of degree 3. Hence the minimal degree of such polynomials p(X,Y) is 3.
(b) Let p(X,Y) be a nonzero polynomial of degree 3 with integer coefficients such that p(a,b)=0 for every element (a,b) in S. From (a), there exists a0∈{1,2,3,4} such that p(a0,Y)=0. Let p0(X,Y)=(X+Y−5)((2X−5)2+(2Y−5)2−10) and define the polynomial
q(X,Y)=p0(a0,a0)p(X,Y)−p(a0,a0)p0(X,Y).
Polynomial q(X,Y) is of degree at most 3. But polynomial q(a0,Y) has at least 1,2,3,4 as roots. Therefore q(a0,Y)=0, which means that q(X,Y)=(X−a0)q1(X,Y) for some polynomial q1(X,Y) of degree at most 2. But for any a∈{1,2,3,4}∖{a0},q1(a,Y) has at least 3 roots, which implies that q1(a,Y)=0. Hence (X−1)(X−2)(X−3)(X−4) divides q(X,Y) of degree 3. Therefore q(X,Y)=0. Because p0(X,Y) is a unitary polynomial, we deduce that
p(X,Y)=kp0(X,Y), for some k∈Z.