For n≥1, denote the n-th composition of f with itself by
fn=deff∘f∘⋯∘f.
By hypothesis, if g∈A satisfies f∘g∘f=g∘f∘g, then g=f. A natural idea is to try to plug in g=fn for some n in the expression f∘g∘f=g∘f∘g in order to get fn=f, which solves the problem:
Claim: If there exists n≥3 such that fn+2=f2n+1, then the restriction f:T→T of f to T is a bijection.
Proof. Indeed, by hypothesis,
fn+2=f2n+1⇔f∘fn∘f=fn∘f∘fn⇒fn=f.
Since n−2≥1, the image of fn−2 is contained in T=f(S), hence fn−2 restricts to a function fn−2:T→T. This is the inverse of f:T→T. In fact, given t∈T, say t=f(s) with s∈S, we have
t=f(s)=fn(s)=fn−2(f(t))=f(fn−2(t)).
i.e., fn−2∘f=f∘fn−2=id on T
(here id stands for the identity function). Hence, the restriction f:T→T of f to T is bijective with inverse given by fn−2:T→T.
It remains to show that n as in the claim exists. For that, define
Sm=deffm(S)(Sm is the image of fm).
Clearly the image of fm+1 is contained in the image of fm, i.e., there is a descending chain of subsets of S
S⊇S1⊇S2⊇S3⊇S1⊇…
which must eventually stabilise since S is finite, i.e., there is a k≥1 such that
Sk=Sk+1=Sk+2=Sk+3=…=defS∞.
Hence f restricts to a surjective function f:S∞→S∞, which also bijective since S∞⊆S is finite. To sum up, f:S∞→S∞ is a permutation of the elements of the finite set S∞, hence there exists an integer r≥1 such that fr=id on S∞ (for example, we may choose r=∣S∞∣!). In other words,
fm+r=fm on S for all m≥k.(1)
Clearly, Eq. (1) also implies that fm+tr=fm for all integers t≥1 and m≥k. So, to find n as in the claim and finish the problem, it is enough to choose m and t in order to ensure that there exists n≥3 satisfying
{2n+1=m+trn+2=m⇔{m=3+trn=m−2.
This can be clearly done by choosing m large enough with m≡3(modr). For instance, we may take n=2kr+1, so that
fn+2=f2kr+3=f4kr+3=f2n+1
where the middle equality follows by Eq. (1) since 2kr+3≥k.