Solution 1. First note that f(0)=c, f(1)=a+b+c, f(−1)=a−b+c. Second, note that if f(α)=f(β), then a(α2−β2)+b(α−β)=0, so (α−β)(a(α+β)+b)=0. Third, a key observation is that for fixed k, there are at most two distinct real numbers x for which f(x)=k. It follows from this that since f(f(1))=f(f(0))=f(f(−1)), the set {f(1),f(0),f(−1)} can contain at most two distinct numbers. This leads us to three cases.
Case 1: f(1)=f(−1).
This gives b=0, so f(1)=f(−1)=a+c and f(0)=c. Therefore
0=f(f(1))−f(f(0))=(f(1)−f(0))(a(f(1)+f(0))+0)=a2(a+2c).
So c=−a/2, which yields f(x)=a(x2−21). Let us verify that it works. First, f(0)=−2a and f(1)=f(−1)=2a. But f is an even function, so f(−2a)=f(2a) and hence f(f(1))=f(f(0))=f(f(−1)) as required.
Case 2: f(1)=f(0).
This gives b=−a, so f(1)=f(0)=c and f(−1)=2a+c. Therefore
0=f(f(−1))−f(f(1))=(f(−1)−f(1))(a(f(−1)+f(1))+b)=2a(a(2a+2c)−a)=2a2(2a+2c−1).
So 2a+2c−1=0, and hence c=−a+21. This yields f(x)=a(x2−x−1)+21. Let us verify that it works. Note that f(21−x)=f(21+x). Also f(0)=21−2a=f(1) and f(−1)=21+2a. So f(f(1))=f(f(0))=f(f(−1)) as required.
Case 3: f(−1)=f(0).
This gives b=a, so f(−1)=f(0)=c and f(1)=2a+c. Therefore
0=f(f(1))−f(f(−1))=(f(1)−f(−1))(a(f(1)+f(−1))+b)=2a2(2a+2c+1).
So 2a+2c+1=0 and hence c=−a−21. This yields f(x)=a(x2+x−1)−21. Let us verify that it works. Note that f(−21−x)=f(−21+x). Also f(0)=−2a−21=f(−1) and f(1)=2a−21. So f(f(1))=f(f(0))=f(f(−1)) as required.
In summary, the solutions are
a(x2−21),a(x2−x−1)+21,a(x2+x−1)−21
Solution 2. First note that f(f(x)) is a quartic. The quartic g(x)=ax4+βx3+γx2+δx+ϵ has the property that g(1)=g(0)=g(−1) if and only if α+γ=0 and β+δ=0. For the given function
f(f(x))=a(ax2+bx+c)2+b(ax2+bx+c)+c
So the property f(f(1))=f(f(0))=f(f(−1)) gives a3+ab2+2a2c+ab=0 and 2a2b+2abc+b2=0.
Since a=0, the first equation gives b2+b+a2+2ac=0. The latter equation gives b(2a2+2ac+b)=0. So b=0 or b=−2a2−2ac=−2a(a+c). We look at these cases separately.
Case 1: b=0.
Then a2+2ac=0 and hence c=−a/2 (since a=0). Then f(x)=a(x2−21). Let us verify that it works. First, f(0)=−2a and f(1)=−21. But f is even, so f(−21)=f(21) and hence f(f(1))=f(f(0))=f(f(−1)) as required.
Case 2: b=−2a2−2ac=−2a(a+c).
Then the first equation gives 4a2(a+c)2−2a2−2ac+a2+2ac=0. Since a=0, this gives (a+c)2=1/4. Thus c=−a±21 and hence b=∓a. So we get two solutions f(x)=a(x2−x−1)+21 and f(x)=a(x2+x−1)−21, for any a=0.
We check that both of these work. For the first function, f(21−x)=f(21+x). Also, f(0)=21−21=21 and f(−1)=21+21. So f(f(1))=f(f(0))=f(f(−1)) as required.
For the second function, f(−21−x)=f(−21+x). Also, f(0)=−21−21=f(−1) and f(1)=21−21. So f(f(1))=f(f(0))=f(f(−1)) as required.
In summary, the solutions are
a(x2−21),a(x2−x−1)+21,a(x2+x−1)−21,
for any a=0.