For simplicity write f2(x)=f(f(x)), f3(x)=f(f(f(x))), etc. We first show
f2(x)=xfor all x>0
in two different ways.
Method 1, using injectivity. Suppose f(z1)=f(z2). Replacing z by z1 or by z2 leads to the same LHS of (2), hence the RHS must agree in both cases as well, i.e. z1+f(y)+f2(x)=z2+f(y)+f2(x) which implies z1=z2, hence f must be injective.
We make the RHS of (2) equal to z+f2(y)+f2(x) in two different ways, in order to benefit from the injectivity of f. Replacing y by f(y) gives f(x+f2(y)+f2(z))=z+f2(y)+f2(x). Replacing x by y and y by f(x) gives f(y+f2(x)+f2(z))=z+f2(x)+f2(y). Because the RHS is the same in both equations, and f is injective, it follows that x+f2(y)+f2(z)=y+f2(x)+f2(z), i.e. f2(x)−x=f2(y)−y for all positive x,y.
This means that f2(x)−x=c is a constant that does not depend on x. Setting t=1+f(1)+f2(1), equation (2) with x=y=z=1 implies f(t)=t, hence f2(t)=t and f2(t)−t=0. Therefore c=0 and we have shown f2(x)=x for all x>0.
Method 2, using fixed points. (Tianci Yan) Putting x=z in (2) gives
f(x+f(y)+f(f(x)))=x+f(y)+f(f(x))
and this means that x+f(y)+f(f(x)) is a fixed point for all x,y>0.
Let a be any fixed point of f. Putting x=y=a in (2) gives
f(2a+f(f(z)))=z+2a, for z>0.
Putting y=z=a in (2) gives
f(x+2a)=2a+f(f(x)), for x>0.
Applying f to both sides of the previous equation and using the earlier result gives
f(f(x+2a))=x+2a, for x>0.
Replacing x by x+2a in the previous equation then gives
f(x+4a)=2a+f(f(x+2a))=x+4a, for x>0.
So f(w)=w, for all integers w>4a. Pick such an integer w>4a and put y=z=w in the original equation (2) to get
f(x+f(w)+f(f(w)))=w+f(w)+f(f(x))
which can be simplified to x+2w=f(x+2w)=2w+f(f(x)) for x>0.
This shows that f(f(x))=x for all x>0, which is f2(x)=x.
Having established f2(x)=x, we can complete the solution of the problem as follows. Given an integer n≥3, we set x=n−2, y=f(1) and z=1 in (2). With the aid of f2(x)=x this becomes f(n)=n for all n≥3. Because of f2(x)=x, we then must have f(1)<3 and f(2)<3. Again from f2(x)=x we see that f(1)=2 implies f(2)=1 and vice versa. This leaves us with two possible solutions:
f1(n)f2(n)=n for n≥0=n for n≥3,f2(1)=2,f2(2)=1.
Clearly, f1 satisfies (2). To verify the functional equation for f2 we first note that x+f2(y)+f22(z)≥3, hence (2) is satisfied if and only if
x+f2(y)+f22(z)=z+f2(y)+f22(x),
i.e. f22(x)−x=f22(z)−z for all x,z∈Z+. Clearly f22(z)−z=0 for z≥3. Since f2(f2(1))=f2(2)=1 and f2(f2(2))=f2(1)=2 we see that indeed f22(x)−x=0 for all x∈Z+. This shows that f2 is a solution as well.