Solution:
We observe that if u≤v then by replacing (u,v) with (u−ε,v+ε), where ε>0, the sum of squares increases. Indeed,
(u−ε)2+(v+ε)2−u2−v2=2ε(v−u)+2ε2>0
Then, denoting
E(a,b,c,d,x,y,z,t)=a2+b2+c2+d2+x2+y2+z2+t2
and assuming without loss of generality that a≤b≤c≤d and x≤y≤z≤t, we have
E(a,b,c,d,x,y,z,t)≤E(0,0,0,0,a+x,b+y,c+z,d+t)≤E(0,0,0,0,1,b+y,c+z,a+d+x+t−1)≤E(0,0,0,0,1,1,c+z,a+b+d+x+y+t−2)≤E(0,0,0,0,1,1,1,5)=28
Note that if (a,b,c,d,x,y,z,t)=(0,0,0,0,1,1,1,5), at least one of the above inequalities, obtained by the ϵ replacement mentioned above, should be a strict inequality. Thus, the maximum value of E is 28, and it is obtained only for (a,b,c,d,x,y,z,t)=(0,0,0,0,1,1,1,5) and permutations of a,b,c,d and of x,y,z,t.
Alternative solution by PSC.
Since 0≤a,b,c,d≤1 we have that a2≤a,b2≤b,c2≤c and d2≤d. It follows that
a2+b2+c2+d2≤a+b+c+d
Moreover, using the fact that y+z+t≥3, we get that x≤5. This means that
(x−1)(x−5)≤0⟺x2≤6x−5
Similarly we prove that y2≤6y−5,z2≤6z−5 and t2≤6t−5. Adding them we get
x2+y2+z2+t2≤6(x+y+z+t)−20
Adding (1) and (2) we have that
a2+b2+c2+d2+x2+y2+z2+t2≤a+b+c+d+6(x+y+z+t)−20≤6(a+b+c+d+x+y+z+t)−20=28
We can readily check that the equality holds if and only if (a,b,c,d,x,y,z,t)=(0,0,0,0,1,1,1,5) and permutations of a,b,c,d and of x,y,z,t.