a) f(x)=−1 or f(x)=x+1.
(Alternative solution by I. Voronovich)
Setting y=f(x) in the given equation
f(x−f(y))=f(f(x))−f(y)−1,(1)
we obtain f(x−f(f(x)))=−1, i.e., there exists an integer λ such that f(λ)=−1. Set y=λ in (1), then
f(f(x))=f(x+1).(2)
Substituting f(x)+f(y) for x in (1), we obtain
f(f(x))=f(f(x)+f(y))−f(y)−1.
Due to symmetry
f(f(y))=f(f(y)+f(x))−f(x)−1.
Therefore f(f(x))−f(x)=f(f(y))−f(y), so f(f(x))−f(x)=c or, in view of (2),
f(x+1)−f(x)=c,c∈Z(3)
If c=0, then f(x+1)=f(x) for all integer x, so f(x) is a constant function, thus from (1) it follows that c=−1, i.e., f(x)=−1 for all integer x.
If c>0 or c<0, then we conclude from (3) that f(x) is increasing or decreasing, respectively. Anyway, f is injective, so (2) gives f(x)=x+1.
Both the functions obviously satisfy the initial equation (1).
b) f(x)=−2 or f(x)=x+2.
(Solution by A. Asanau.) First, in the same way as in a) we can easily establish for any f:Z→Z satisfying the given equation
f(x−f(y))=f(f(x))−f(y)−2(1)
the following equalities:
f(x−f(f(x)))=−2,(2)
f(f(x))=f(x+2),(3)
f(x+2)−f(x)=c,(4)
where c is a constant. In particular, (2) shows that −2∈E(f), the range of f. Now we have two cases.
I. c=0. Let f(0)=a, f(1)=b. Then from (4) we have f(x+2)=f(x), so f(x)=a for all even x and f(x)=b for all odd x. Note that one of a, b equals −2 in view of (2). We claim that a=b=−2. Suppose that a=b. Then
f(x)=f(y)⟺x≡y(mod2).
Therefore (3) implies f(x)≡x+2≡x(mod2). In particular, f(0)≡0(mod2), whence f(0)=a=−2.
Now we set x=0, y=1 in (1); then we get f(−b)=f(−2)−b−2=−b−4. But f(−b) equals either b or −2. In both cases we have b=−2, contrary to a=b.
So a=b=−2, and f(x)=−2 for all x. It is easy to see that this function satisfies the initial equation.
II. c=0. Let again f(0)=a, f(1)=b. Then from (4) we have
f(2n)=cn+a,f(2n+1)=cn+b.(5)
In particular, (5) implies that, on the one hand, f(x)→∞ as x→∞, and, on the other hand, limx→∞xf(x)=2c. Then from (3) we have
f(x)f(f(x))=f(x)f(x+2)→1 as x→∞,
but f(x)f(f(x))→2c. Hence, c/2=1, i.e. c=2. Then (5) becomes
f(x)f(x)=x+a=x+dfor even x,for odd x(where d=b−1).(6)
1) Now, if f is an injective function, then (3) immediately implies f(x)=x+2, and this function satisfies (1).
2) Suppose that f is not injective. But from (6) it easily follows that f is injective on the set of even numbers and also injective on the set of odd numbers. Thus we see that some f(2n) and f(2m+1) are equal. From (6) it follows that 2n−2m−1=d−a, i.e., a and d are of different parity. Then (6) implies that all values of f are of the same parity. Since −2∈E(f), all the values of f(x) are even. Then from (3) we have f(x+2)=f(f(x))=f(x)+a. Thus (4) gives a=c=2. In particular, d is odd. So, (6) becomes
f(x)f(x)=x+2=x+dfor even x,for odd x.
One can verify that for any odd d this function is a solution of (1).