Assume that there exists a function $f(x)$ satisfying the problem condition:
$$\begin{cases} \{f(x)\} \sin^2 x + \{x\} \cos f(x) \cos x = f(x), \\ f(f(x)) = f(x), \end{cases}$$
for all real $x$.
Replacing $x$ by $f(x)$ in the first equality, we obtain
$$\{f(f(x))\} \sin^2 f(x) + \{f(x)\} \cos^2 f(x) = f(f(x)).$$
Since $f(f(x)) = f(x)$, from the obtained equality it follows that $\{f(x)\} = f(x)$. So $f: \mathbb{R} \rightarrow [0; 1]$.
Replacing $x$ by $\pi$, we have $-\{\pi\} \cos f(\pi) = f(\pi)$. Since $f(\pi) \in [0, 1]$ and $\{\pi\} \neq 0$, we see that the left-hand side of the last equality is negative, whereas the right-hand side is nonnegative, a contradiction.