Solution:
We show that the solution set consists of {(t,0,0);t∈Z}∪{(−1,−1,1)}. Let us put a+b+c=d, ab+bc+ca=e and abc=f. The given condition f(f(a,b,c))=(a,b,c) implies that
d+e+f=a,de+ef+fd=b,def=c
Thus abcdef=fc and hence either cf=0 or abde=1.
Case I: Suppose cf=0. Then either c=0 or f=0. However c=0 implies f=0 and vice-versa. Thus we obtain a+b=d, d+e=a, ab=e and de=b. The first two relations give b=−e. Thus e=ab=−ae and de=b=−e. We get either e=0 or a=d=−1.
If e=0, then b=0 and a=d=t, say. We get the triple (a,b,c)=(t,0,0), where t∈Z. If e=0, then a=d=−1. But then d+e+f=a implies that −1+e+0=−1 forcing e=0. Thus we get the solution family (a,b,c)=(t,0,0), where t∈Z.
Case II: Suppose cf=0. In this case abde=1. Hence either all are equal to 1; or two equal to 1 and the other two equal to −1; or all equal to −1.
Suppose a=b=d=e=1. Then a+b+c=d shows that c=−1. Similarly f=−1. Hence e=ab+bc+ca=1−1−1=−1 contradicting e=1.
Suppose a=b=1 and d=e=−1. Then a+b+c=d gives c=−3 and d+e+f=a gives f=3. But then f=abc=1⋅1⋅(−3)=−3, a contradiction. Similarly a=b=−1 and d=e=1 is not possible.
If a=1,b=−1,d=1,e=−1, then a+b+c=d gives c=1. Similarly f=1. But then f=abc=1⋅1⋅(−1)=−1 a contradiction. If a=1,b=−1,d=−1,e=1, then c=−1 and e=ab+bc+ca=−1+1−1=−1 and a contradiction to e=1. The symmetry between (a,b,c) and (d,e,f) shows that a=−1,b=1,d=1,e=−1 is not possible. Finally if a=−1,b=1,d=−1 and e=1, then c=−1 and f=−1. But then f=abc is not satisfied.
The only case left is that of a,b,d,e being all equal to −1. Then c=1 and f=1. It is easy to check that (−1,−1,1) is indeed a solution.
Alternatively
cf=0 implies that ∣c∣≥1 and ∣f∣≥1. Observe that
d2−2e=a2+b2+c2,a2−2b=d2+e2+f2
Adding these two, we get −2(b+e)=b2+c2+e2+f2. This may be written in the form
(b+1)2+(e+1)2+c2+f2−2=0
We conclude that c2+f2≤2. Using ∣c∣≥1 and ∣f∣≥1, we obtain ∣c∣=1 and ∣f∣=1, b+1=0 and e+1=0. Thus b=e=−1. Now a+d=d+e+f+a+b+c and this gives b+c+e+f=0. It follows that c=f=1 and finally a=d=−1.