Let (a,b,c) be a solution of the system. If one of the numbers a,b,c is 0, e.g. if c=0, then c(a2+b)=a(b+ca) leads to a=0, and similarly one gets b=0. Thus abc=0 leads to a=b=c=0 which is indeed a solution. We now look for solutions with abc=0. We rewrite the equations
ab(b−c)bc(c−a)ca(a−b)=c(c−a)=a(a−b)=b(b−c).
It follows that a2b2c2(a−b)(b−c)(c−a)=abc(a−b)(b−c)(c−a).
Case 1: Among the numbers a,b,c there exist (at least) two equal ones; say a=b. We have
a2c+bc=b2+abc⇔bc=b2⇔c=b,
therefore in this case we obtain a=b=c. Conversely, any such triple is a solution to the system.
Case 2: If a=b=c=a, then abc=1. We obtain ab(b−c)=c(c−a)⇔(b−c)=c2(c−a) and two more equations, similar to this one. It follows that
a3+b3+c3=ac2+ba2+cb2.
If a,b,c>0, from AM-GM it follows that a3+a3+b3≥3ba2 (with equality if and only if a=b) which, added with two similar inequalities, leads to a3+b3+c3≥ac2+ba2+cb2. We thus have equality in the previous inequality, hence a=b=c.
If one of the variables is positive and the other two are negative, say a>0, b,c<0, then
b(c2+a)=a(a+bc)=a2+1>0⇒a<0,
contradiction.
In conclusion, the only solutions are (a,b,c)=(x,x,x), ∀x∈R.