(Solution by A. Asanau, Y. Dubovik, Y. Laurenau, V. Vityaz.) Suppose that such a, b, f exist. Obviously b=0. Further, since f is surjective, there exists a λ such that f(λ)=0. Set x=λ in
f(f(x))=bxf(x)+a,(1)
then f(0)=f(f(λ))=bλf(λ)+a=bλ⋅0+a=a. Further,
f(a)=f(f(0))=b⋅0⋅f(0)+a=a.
Now a=f(a)=f(f(a))=baf(a)+a=ba2+a whence a=0. Thus (1) becomes
f(f(x))=bxf(x).(2)
Further, there exists a σ=0 such that f(σ)=−1/b. Set x=σ in (2), then
f(−1/b)=f(f(σ))=bσf(σ)=bσ(−1/b)=−σ.(3)
Hence
f(−σ)=f(f(−1/b))=b(−1/b)f(−1/b)=σ.
Therefore,
b1=f(σ)=f(f(−σ))=b(−σ)f(−σ)=−bσ2
and so σ=±1/b.
If σ=−1/b, then σ=−1/b=f(σ)=f(−1/b), contrary to (3).
If σ=1/b, then in view of (3) we have
f(−σ)=−σ⟹−σ=f(−σ)=f(f(−σ))=b(−σ)f(−σ)=bσ2⟹σ=−1/b,
a contradiction.
Therefore, there are no a, b, f satisfying the problem condition.