The minimum value of a2+b2 is 54.
WLOG assume a≥0. (Indeed, by flipping the sign of a, we only need to flip the sign of the root x.) Clearly, x=0. Rewrite the equation as
x2+ax+b+xa+x21=0.
This is the same as
(x+x1)2+a(x+x1)+b−2=0.
Let y=x+x1. Note that ∣y∣=∣x∣+∣x∣1≥2 by the AM-GM inequality. The equation becomes
y2+ay+(b−2)=0.
If a>1, then a2+b2>1. Thus, we only need to consider the case a≤1. This implies the axis of symmetry y=−2a lies between −2 and 2.
* If y≥2, then we have
0=y2+ay+(b−2)≥(2)2+a(2)+(b−2)=2a+b+2.
This yields b≤−2a−2≤−2, and hence a2+b2≥4.
* If y≤−2, then we have
0=y2+ay+(b−2)≥(−2)2+a(−2)+(b−2)=−2a+b+2.
This yields b≤2a−2≤0, and hence
a2+b2≥a2+(2a−2)2=5(a−54)2+54≥54.
Equality holds when a=54 and b=−52. Correspondingly, this means y=−2 and x=−1.
To conclude, the minimum value of a2+b2 is 54.