Answer: f(x)=x, f(x)=0.
Let us denote the given equation by (1):
f(x+yf(x+y))=f(y2)+xf(y)+f(x).(1)
Substitute x=y=0 into (1):
f(0)=2f(0)⟹f(0)=0.
Substitute x=0 into (1):
f(yf(y))=f(y2).(2)
Substitute y=−x into (1):
f(x+(−x)f(x−x))=f(x2)+xf(−x)+f(x)⟹f(x)=f(x2)+xf(−x)+f(x)
So,
f(x2)=−xf(−x).(3)
Now, substitute x→−x in (3):
f(x2)=−(−x)f(−(−x))=xf(x)
So,
f(x2)=xf(x)
Comparing with (3):
xf(x)=−xf(−x)⟹f(−x)=−f(x).(4)
So f is odd.
Now, substitute y=−y in (1):
f(x+(−y)f(x−y))=f(y2)+xf(−y)+f(x)
But f(−y)=−f(y), so
f(x+(−y)f(x−y))=f(y2)−xf(y)+f(x)
Let us write this as:
f(x+yf(y−x))=f(y2)−xf(y)+f(x).(5)
Now, consider the difference between (1) and (5):
f(x+yf(x+y))−f(x+yf(y−x))=2xf(y).(6)
Suppose that for some x0=0, f(x0)=0. Substitute y=x0−x into (1):
f(x+(x0−x)f(x+x0−x))f(x)f((x0−x)2)=f((x0−x)2)+xf(x0−x)+f(x)=f((x0−x)2)+xf(x0−x)+f(x)=−xf(x0−x)
But from (3):
f((x0−x)2)=−(x0−x)f(x−x0)=(x0−x)f(x0−x)
So,
−xf(x0−x)=(x0−x)f(x0−x)⟹f(x0−x)=0,∀x∈R.
Thus, f is identically zero, f(x)=0 for all x.
If f is not identically zero, then such x0=0 cannot exist. Thus, from the condition f(x0)=0 it follows that x0=0.
Now, let us prove injectivity. Suppose ∃x1=x2 such that f(x1)=f(x2).
In (6), substitute x=2x1−x2, y=2x1+x2:
f(2x1−x2+2x1+x2f(x1))−f(2x1−x2+2x1+x2f(x2))(x1−x2)f(2x1+x2)=(x1−x2)f(2x1+x2)=0
So, either x1=x2 (contradicts assumption), or f(2x1+x2)=0. But as above, the only zero is at 0, so x1+x2=0⟹x1=−x2.
But then
f(x2)=f(−x1)=−f(x1)=−f(x2)⟹f(x2)=0⟹x2=0⟹x1=0
So, f is injective.
Now, from (2):
f(yf(y))=f(y2)
But f is injective, so yf(y)=y2⟹f(y)=y for all y.
Thus, the only solutions are f(x)=0 and f(x)=x.
It is easy to check that both satisfy the original equation.