IRL ABooklet

Letting $x = 0, y = 1$ we get $f(1)f(0) = f(0)$, so $f(0) = 0$ or $f(1) = 1$.

Case 1: $f(0) = 0$
Letting $y = 0$ and $x = 1 - t$ we get $f(t) = 0$ in this case for all $t$. Indeed, this is a valid solution for our equation.

Case 2: $f(0) \neq 0$
In this case we have $f(1) = 1$. We first show that $f(x) \neq 0$ for all $x$. Suppose $f(x) = 0$ for some $x \neq 0, 1$, then
$$f\left(\frac{x+y-1}{xy-1}\right) = 0 \quad \text{for all } y \neq \frac{1}{x}. \qquad (6)$$
But note that for any $t \neq 1/x$,
$$t = \frac{x+y-1}{xy-1} \iff y = \frac{x+t-1}{xt-1}$$
so by giving this value of $y$ in equation (6) we get $f(t) = 0$ for all $t \neq 1/x$ provided that $\frac{x+t-1}{xt-1} \neq \frac{1}{x}$. In particular, when $t = 1$ we have $t \neq 1/x$ and $\frac{x}{x-1} \neq \frac{1}{x}$ because $x^2 - x + 1 = (x - \frac{1}{2})^2 + \frac{3}{4} > 0$, hence $f(1) = 0$, which is not true in the current case. Therefore $f(x) \neq 0$ for all $x$.
Letting $y = 0$ in the original equation we get $f(x)f(0) = f(1-x)$. Hence for $x = \frac{1}{2}$ we get $f(\frac{1}{2})f(0) = f(\frac{1}{2})$ and so $f(0) = 1$ since $f(\frac{1}{2}) \neq 0$.

One way to find an explicit expression for $f$ starts with the observation that $f(0) = 1$ and $f(x)f(0) = f(1-x)$ imply $f(x) = f(1-x)$ for all $x$.
For any $x$ we let $y = 1 - x$ in the original equation, which is possible since $xy - 1 = -(x^2 - x + 1) \neq 0$, and get
$$f(x)^2 = (x^2 - x + 1)^2$$
which means that $f(x) = \pm(x^2 - x + 1)$ for all $x$, where the choice of sign may depend on $x$. If $x \neq \pm 1$ then letting $y = x$ in the original equation we get
$$f(x)^2 = (x^2 - 1)^2 f\left(\frac{2x-1}{x^2-1}\right) \quad \text{hence} \quad f\left(\frac{2x-1}{x^2-1}\right) \ge 0 \quad \forall x \ne \pm 1.$$
We then note that $y = \frac{2x-1}{x^2-1}$ is equivalent to $x^2y - 2x - y + 1 = 0$, or $x = \frac{1 \pm \sqrt{y^2 - y + 1}}{y}$. Notice that for every $y$, we have $y^2 - y + 1 > 0$ and for each $y \neq 0$ there is an $x \neq \pm 1$ satisfying the relation above. Hence $f(y) \ge 0$ for all $y \neq 0$, which implies $f(y) = y^2 - y + 1$.

A second way to find this formula for $f$ starts with any $x \neq \pm 1$ so that we can define
$$y = \frac{2x - 1}{x^2 - 1}. \qquad (7)$$
We then have $y(x^2 - 1) = 2x - 1$ which implies $y^2x^2 - 2xy = y^2 - y$ after multiplication by $y$. This can be rewritten as
$$(xy - 1)^2 = y^2 - y + 1. \qquad (8)$$
Moreover, using expression (7) for $y$ we obtain
$$\frac{x+y-1}{xy-1} = \frac{(x-1)(x^2-1) + 2x-1}{x(2x-1) - (x^2-1)} = \frac{x(x^2-x+1)}{x^2-x+1} = x$$
because $x^2 - x + 1 > 0$ for all $x$. Therefore, for any $x \neq \pm 1$ and $y$ given by (7), the original equation becomes
$$f(x)f(y) = (y^2 - y + 1)f(x).$$
Since $f(x) \neq 0$, we obtain $f(y) = y^2 - y + 1$ for each $y$ of the form (7). Finally, given $y \neq 0$ we can use (8) to find
$$x = \frac{1 \pm \sqrt{y^2 - y + 1}}{y}$$
such that $x$ and $y$ are related by (7). This means that each non-zero $y$ can be expressed in the form (7), hence $f(y) = y^2 - y + 1$ for all $y \neq 0$. As we know that $f(0) = 1$, this equation holds for all $y$.
Finally we check that the function $f(x) = x^2 - x + 1$ satisfies the original functional equation.
$$$$
$$\begin{align*} f(x)f(y) &= (x^2 - x + 1)(y^2 - y + 1) \\ &= (xy - 1)^2 - (xy - 1)(x + y - 1) + (x + y - 1)^2 \\ &= (xy - 1)^2 \left( \left( \frac{x+y-1}{xy-1} \right)^2 - \left( \frac{x+y-1}{xy-1} \right) + 1 \right) \\ &= (xy - 1)^2 f \left( \frac{x+y-1}{xy-1} \right). \end{align*}$$