, 2015

Let $f$ and $g$ be functions satisfying the functional equation
$$f(x-y) = f(x)f(y) + g(x)g(y) \quad (1)$$
Taking $x \to y$ in (1), we get
$$f(0) = f(x)^2 + g(x)^2. \quad (2)$$
Taking $y \to 0$ in (1), we get
$$f(x)(1-f(0)) = g(x)g(0). \quad (3)$$
Combining (2) and (3), we get
$$\begin{aligned} f(0)(1-f(0))^2 &= (f(x)^2 + g(x)^2)(1-f(0))^2 \\ &= g(x)^2(g(0)^2 + (1-f(0))^2). \end{aligned}$$
Since $g$ is non-constant, we have
$$f(0) = 1 \quad \text{and} \quad g(0) = 0.$$
Hence
$$f(x)^2 + g(x)^2 = 1$$

$$f(-x) = f(x). \quad (4)$$

Taking $y \to -y$ in (1), we get
$$\begin{aligned} f(x+y) &= f(x)f(-y) + g(x)g(-y) \\ &= f(x)f(y) + g(x)g(-y). \end{aligned}$$
It follows that
$$g(x)g(-y) = g(-x)g(y) \quad (5)$$
for all $x, y \in \mathbb{R}$. Since $f$ is non-constant, there exists $u, v \in \mathbb{R}$ such that $f(u-v) \neq f(u+v)$. Then, clearly $g(v) \neq g(-v)$. On the other hand, we have $g(v)^2 = 1 - f(v)^2 = 1 - f(-v)^2 = g(-v)^2$, hence $g(-v) = -g(v) \neq 0$. From (5), we see that
$$g(-x) = -g(x).$$

a.
From (4) and (5), we have
$$f(x+y) = f(x)f(y) - g(x)g(y). \quad (6)$$
Consequently, we have
$$\begin{aligned} f(x) &= f(x+y-y) \\ &= f(x+y)f(y) + g(x+y)g(y) \\ &= f(x)f(y)^2 - g(x)g(y)f(y) + g(x+y)g(y) \\ &= f(x)(1-g(y)^2) - g(x)f(y)g(y) + g(x+y)g(y) \\ &= f(x) - g(y)(g(x+y) - f(x)g(y) - g(x)f(y)), \end{aligned}$$
hence
$$g(y)(g(x+y) - f(x)g(y) - g(x)f(y)) = 0.$$
It follows that for any $y$ with $g(y) \neq 0$, we have
$$g(x+y) = f(x)g(y) + g(x)f(y). \quad (7)$$
Exchanging $x$ and $y$ we see that (7) holds if $g(x) \neq 0$. If $g(x) = g(y) = 0$, then $f(x)^2 = f^2(y) = 1$, thus $f(x+y)^2 = 1$ by (6). Hence $g(x+y) = 0$ and thus (7) also holds. This proves (a).

b.
Let $h: \mathbb{R} \to \{z \in \mathbb{C} \mid |z| = 1\}$ denote the function given by $h(x) = f(x) + ig(x)$. Then it follows from (6) and (7) that
$$h(x+y) = h(x)h(y).$$
Since $h$ is continuous, we see that there exists $c \in \mathbb{R}$ such that $h(x) = \exp(icx)$. Hence
$$\begin{cases} f(x) = \cos(cx) \\ g(x) = \sin(cx). \end{cases} \quad (8)$$
Conversely, it is clear that for any $c \in \mathbb{R}$, (8) gives a solution to the problem.