Suppose that there exists a function f:R→R satisfying
f(f(x))=1−xf(x)(∗)
for any x∈R.
Set c=f(0). Using (∗) for x=0, we obtain f(c)=f(f(0))=1−0⋅f(0)=1−0⋅c=1.
Since
1=f(c),(1)
we have f(1)=f(f(c))=1−c⋅f(c)=1−c⋅1=1−c. Similarly, since
1−c=f(1),(2)
we have
f(1−c)=f(f(1))=1−f(1)=1−(1−c)=c.(3)
It follows that f(c)=f(f(1−c))(∗)=1−(1−c)⋅f(1−c)(3)=1−(1−c)c=c2−c+1, i.e.
c2−c+1=f(c).(4)
From (1) and (4) it follows that c2−c+1=1, so either c=0 or c=1.
We show that both the equalities c=0 and c=1 give a contradiction. Indeed, if c=0, then from definition of c it follows that 0=f(0), but (1) gives 1=f(0), a contradiction. If c=1, then (1) and (2) give 1=f(1) and 0=f(1) respectively, a contradiction. Therefore there is no function satisfying (∗).