Silk Road Mathematical Competition

Answer. The answer is no.

Let $f(x) = 1 + \frac{1}{x}$ and $f^{(n)}(x) = f(f(\dots f(x)))$ (where $f$ is applied $n$ times). For each $x \in S$ the set $S$ contains all numbers of the form $f^{(n)}(x)$. If $S$ is finite, then $f^{(k)}(x) = f^{(\ell)}(x)$ for some $k > \ell$. Since $f(a) = f(b)$ implies $a = b$, $f^{(k-\ell)}(x) = x$. Thus each $x \in S$ is a root of equation $f^{(n)}(x) = x$ for some positive integer $n$.

The functions $f^{(n)}(x)$ obviously are of the form $\frac{px+q}{rx+s}$ with some $p, q, r, s$ (it is immediate for $n=1$, and $f$ transforms every such function into a function of this form). Therefore, the equation $f^{(n)}(x) = \frac{px+q}{rx+s} = x$ is reduced to an equation of the form $rx^2 + (s-p)x - q = 0$ which is an equation of degree at most 2. This equation is not identical (since, for example, $f(-2) = \frac{1}{2}$ and therefore $f^{(n)}(-2)$ is positive for all positive integers $n$), hence it cannot have more than two real roots. However, the roots $\frac{1+\sqrt{5}}{2}$ and $\frac{1-\sqrt{5}}{2}$ of $f(x) = x$ are also roots of $f^{(n)}(x) = x$ and therefore its only roots.

We see now that if $S$ is finite, it can contain at most two elements.

Second solution. There is an alternative way to prove that for every $x$ except $\varphi = \frac{1+\sqrt{5}}{2}$ and $\bar{\varphi} = \frac{1-\sqrt{5}}{2}$, $f^{(n)}(x)$ and $x$ would not coincide.

If $x < -1$, then $f(x) > 0$ and therefore all $f^{(n)}(x)$ are positive.

If $0 < x \le 1$, then $f(x) > 1$, and all of $f^{(n)}(x)$ are greater than 1.

If $x > 1$, then $|f(x) - \varphi| = |1 + \frac{1}{x} - 1 - \frac{1}{\varphi}| = \frac{|x-\varphi|}{x\varphi} < |x - \varphi|$, that is, the distance from $f^{(n)}(x)$ to $\varphi$ decreases as $n$ increases.

Finally, if $-1 < x < 0$, while the terms of the sequence $x, f(x), f(f(x)), \dots$ remain in the interval $(-1, 0)$, their distances between each term of the sequence and $\bar{\varphi}$ would increase (indeed, $|f(t) - \bar{\varphi}| = |1 + \frac{1}{t} - 1 - \frac{1}{\bar{\varphi}}| = \frac{|t-\bar{\varphi}|}{t\bar{\varphi}} > |t - \bar{\varphi}|$), and when a term appears outside this interval, all the proceeding terms would also be outside of this interval.