Let $P(x,y)$ denote plugging $x$ and $y$ into the original equation. We have
$$\begin{aligned} P(0,1) &: f(0) = 0, \\ P(1,1) &: f(f(1)) = f(1), \\ P(1,f(1)) &: f(f(f(1))) = (1-f(1))f(f(1)) + f(1)^2 f(f(1)). \end{aligned}$$
By combining the second and the third equality we get
$$f(1) = (1 - f(1)) \cdot f(1) + f(1)^3, \quad \text{i.e.} \quad f(1)^2 = f(1)^3.$$
From this we have two options: $f(1) = 1$ or $f(1) = 0$.

If $f(1) = 1$, we get
$$P(x,1) : \quad f(x) = x^2,$$
so $f(x) = x^2$ should hold for all real $x$. By plugging this into the original equation we can conclude that this is not possible. Hence, $f(1) = 0$ must hold.

Let $c$ be an arbitrary root of the function $f$. We have
$$P(x,c) : \quad 0 = (1-c)f(xc).$$
Therefore, if $c \ne 1$, $f(cx) = 0$ must hold for all $x \in \mathbb{R}$. If $c \ne 0$ this implies that $f(x) = 0$ for all $x \in \mathbb{R}$. We check directly that $f \equiv 0$ is indeed a possible solution of the original equation.

Now assume that $f$ is not identically equal to 0. Calculations above show in that case 0 and 1 are the only roots of the function $f$. Now we have
$$P(1,y) : \quad f(f(y)) = (1-y)f(y) + y^2 f(y) = (1-y+y^2)f(y), \quad (1)$$
and for $x \ne 0$
$$P\left(\frac{1}{x}, x\right) : \quad f\left(\frac{f(x)}{x}\right) = f(x), \quad (2)$$
since $f(1) = 0$.

Let $y_1, y_2 \in \mathbb{R} \setminus \{0,1\}$ be such that $f(y_1) = f(y_2) \ne 0$. Then from (1) it follows that
$$1 - y_1 + y_1^2 = 1 - y_2 + y_2^2, \quad \text{i.e.} \quad (y_1 - y_2)(y_1 + y_2 - 1) = 0.$$
We conclude that
$$f(y_1) = f(y_2) \implies y_1 = y_2 \text{ or } y_1 + y_2 = 1. \quad (3)$$
By combining (2) and (3) we get that for all $x \ne 0,1$ we have
$$\frac{f(x)}{x} = x \text{ or } \frac{f(x)}{x} + x = 1,$$
i.e.
$$f(x) = x^2 \text{ or } f(x) = x - x^2.$$
Let $x \in \mathbb{R} \setminus \{0,1\}$ be such that $f(x) = x^2$. By plugging $y = x$ into (1) we get
$$f(x^2) = (1 - x + x^2)x^2.$$
Now we have two cases: $f(x^2) = x^4$ and $f(x^2) = x^2 - x^4$. In the first case, we get $x^4 = (1-x+x^2)x^2$, from which we can easily conclude that $x=0$ or $x=1$. Since we assumed $x \notin \{0,1\}$, this is not possible. In the second case, we get $x^2 - x^4 = (1-x+x^2)x^2$, from which we can conclude $x=0$ or $x = \frac{1}{2}$. Again, we can dismiss $x=0$, and for $x = \frac{1}{2}$ we can note that $f(\frac{1}{2}) = (\frac{1}{2})^2 = \frac{1}{4} = \frac{1}{2} - (\frac{1}{2})^2$. This shows that $f(x) = x - x^2$ holds for all $x \ne 0,1$. Since this formula holds for $x = 0,1$ as well, we can conclude that
$$f(x) = x - x^2, \quad \forall x \in \mathbb{R}.$$
By plugging this directly into the original equation, we can confirm that this is indeed a solution. Therefore, the only solutions of the original equation are $f = 0, \forall x \in \mathbb{R}$ and $f(x) = x - x^2, \forall x \in \mathbb{R}$.