Label the given equations as follows:
a+b+c=1(10)
a2+b2+c2=3(11)
bc−x−2ya=0(12)
ca−x−2yb=0(13)
ab−x−2yc=0.(14)
Solution 1. Eliminate x by subtracting (13) from (12), and (14) from (13) to get c(b−a)=2y(a−b) and a(c−b)=2y(b−c). From these it follows that
c(b−a)(b−c)=2y(a−b)(b−c)=a(c−b)(a−b),
i.e.,
0=a(c−b)(a−b)−c(b−a)(b−c)=(a−b)(b−c)(c−a).
So, one of a−b,b−c,c−a must be zero. Wlog, suppose that a=b. Then, by (10) and (11),
2a+c=1and2a2+c2=3.
Hence 3=2a2+(1−2a)2=6a2−4a+1 and so 0=(a−1)(3a+1). If a=1, we have a=b=1 and c=1−2a=−1 which gives abc=−1. If a=−1/3, we have a=b=−1/3, c=1−2a=5/3 and so abc=5/27.
Solution 2. From (10) and (11) we obtain
−2=(a+b+c)2−(a2+b2+c2)=2(ab+bc+ca), i.e.
ab+bc+ca=−1.(15)
Adding (12), (13) and (14) together, and using (10) and (15) we get
3x+2y=−1.(16)
Multiplying (12), (13) and (14) by a,b,c, respectively, then adding these together and using (10) and (11) gives us
x+6y=3abc.(17)
1=(ab+bc+ca)2=(ab)2+(bc)2+(ca)2+2abc(a+b+c)=(x+2ya)2+(x+2yb)2+(x+2yc)2+2abc=3x2+4xy+12y2+2abc=x(3x+2y)+2y(x+6y)+2abc=−x+6yabc+2abc=6y−3abc+6yabc+2abc=6y(abc+1)−abc.using (15)by (12), (13), (14)using (10), (11)using (16), (17)using (17) again
0=6y(abc+1)−abc−1=(6y−1)(abc+1).
Therefore, either abc=−1, or y=1/6. In the latter case, equation (16)
gives x=−4/9 and then (17) implies abc=5/27.