Solution:
First observe that g(x)=0 and g(x)=x2 are solutions. Indeed, both are even functions. For g(x)=0, one can take f(x)=0 (or any function f such that f(x+f(x))=0). For g(x)=x2, one can take f(x)=x, because (x+y)2=x2+y2+2xy.
We now prove that these are the only solutions. Let g be a solution of the problem and f be an associated function. We start with a bunch of substitutions. Let y=0 which gives
g(f(x))=g(x)+g(0)
Let further y=−f(x) in the original equation and use the parity of g to get g(0)=g(x)+g(f(x))−f(x)f(x+f(x)). If we plug (1) in, we get
2g(x)=f(x)f(x+f(x))
We replace y by f(y) in the original equation and use (1) to obtain g(f(x)+f(y))=g(x)+g(y)+g(0)+f(y)f(x+f(x)). The symmetry between x and y implies
f(y)f(x+f(x))=f(x)f(y+f(y))
If f≡0, then g≡0. So, if we assume that g is not the constant 0 function (which we know is a solution), then there exists a such that f(a)=0. Let c=f(a+f(a))/f(a). We have, with y=a in (3), f(x+f(x))=cf(x) and using (2) we obtain
2g(x)=cf(x)2
Since we assumed g is not identically 0, it holds c=0. We plug (4) in the original equation and use f(x+f(x))=cf(x) to obtain, after simplifying by c,
f(f(x)+y)2=f(x)2+f(y)2+2yf(x)
If one lets x=y in (5), then one obtains
f(x)(f(x)(c2−2)−2x)=0
If c2=2, then f(x)=0 for all x=0. In particular, one must have a=0 and f(0)=f(a)=0. But then c=f(f(0))/f(0)=0 since f(0)=0. Contradiction. Hence c2=2, and for every x,
f(x)=0orf(x)=2x/(c2−2)
In particular, f(0)=0. Let b:=2/(c2−2)=0 such that f(x)=bx if f(x)=0. Since f(a)=0, we have f(a)=ab and a=0. Assume there is t=0 such that f(t)=0, then, with x=a and y=t in (5), we get
f(ab+t)2=a2b2+2tab
Because t=0, we must have f(ab+t)=0 and thus ab+2t=0. So there is at most one t=0, such that f(t)=0, namely t=−ab/2. But if such a t exists, then f(ab+t)=0 and ab+t=t and ab+t=0, contradiction. So, f(x)=bx for all x and g(x)=cb2/2⋅x2 for all x by (4). We plug this in the original equation and deduce that b=1 and c=2 (using that c=0). Therefore, we proved that if g is not identically 0, then g(x)=x2 for every x.