f=⌊x⌋, f=⌈x⌉, and constant functions.
We first verify that the above functions satisfy the problem's conditions. For all triples (x,a,b)∈Q×Z×N, let
q=⌊bx+a⌋.
It is easy to see that bq≤x+a<b(q+1)⇒bq≤⌊x⌋+a<b(q+1), so
⌊b⌊x⌋+a⌋=⌊bx+a⌋.
Thus f=⌊x⌋ is indeed a solution. The other two solutions can be verified similarly.
Next we prove that these are all possible solutions. Consider the following two cases:
Case 1. There exists an integer m such that f(m)=m.
Let C=f(m). If m>C, then for all integers y, substituting x=m,b=m−C,a=yb−C into the original equation gives f(y)=f(y+1), so for all integers y, f(y)=C. If m<C, then instead taking b=C−m, we likewise obtain f(y)=C for all integers y.
Now, for all rational numbers y=p/q (where q>0), substituting (x,a,b)=(C−p,p−C,q) gives f(y)=f(0)=C. Hence f is a constant function.
Case 2. For all integers m, f(m)=m.
Taking b=1, we know that
f(x)+a=f(x+a)(1)
holds for all (x,a)∈Q×Z. Let ω=f(1/2).
Claim 1. ω∈{0,1}.
Proof. If ω≤0, substituting (x,a,b)=(1/2,−ω,1−2ω) gives 0=f(0)=f(1/2)=ω. Similarly, if ω≥1, take (x,a,b)=(1/2,ω−1,2ω−1).
Claim 2. For all rational numbers 0<x<1, f(x)=ω.
Proof. If not, there exists p/q∈(0,1) with the smallest denominator such that f(p/q)=ω. It is easy to see that gcd(p,q)=1, and q≥2. If q is even, then p must be odd, so substituting (x,a,b)=(21,2p−1,2q) gives
f(q/2ω+(p−1)/2)=f(p/q)=ω.
But by Claim 1, ω=0 or 1, so the above equation in either case yields some p′/q′∈(0,1) satisfying f(p′/q′)=ω with q′<q. This contradicts the assumption that p/q has the smallest denominator, a contradiction.
Therefore q must be odd; let q=2k+1. Substituting (x,a,b)=(1/2,k,q) gives
f(qω+k)=f(21)=ω.
Since p,q are coprime, there exist r∈{1,2,…,q} and an integer m such that rp−mq=k+ω; moreover, since the right-hand side is not a multiple of q, r<q. And if m<0, then rp−mq>q≥k+ω, a contradiction, so m≥0. Similarly, if m≥r, then rp−mq<br−br=0. Hence, 0≤m≤r−1. Now, taking (x,a,b)=(qk+ω,m,r), we get
f(rω+m)=f(p/q)=ω.
This contradicts the minimality of q, a contradiction!
Now, if ω=0, then by Claim 2, f(x)=⌊x⌋. Similarly, if ω=1, then by Claim 2, f(x)=⌈x⌉. This completes the proof.