a.
We have f(x)=x3+4x2−4.
Label the equation as follows.
f(x+y)=f(x)+f(y)+axy(x+y)+bxy+c(x+y)+4(1)
Putting x=y=0 in (1), we have
f(0)=f(0)+f(0)+4,
which implies f(0)=−4. Next, putting x=1 and y=0 in (1), we have
f(1)=f(1)+f(0)+c+4.
This yields c=0. Now, putting x=y=1 in (1), we have
f(2)=f(1)+f(1)+2a+b+4.
As it is given that f(1)=1 and f(2)=20, we obtain
2a+b=14.(2)
Also, we put x=y=2 in (1). This gives
f(4)=f(2)+f(2)+16a+4b+4.
This means f(4)=16a+4b+44. Moreover, putting x=4 and y=−4 in (1), we find that
f(0)=f(4)+f(−4)−16b+4.
This becomes
4a−3b=−12.(3)
Solving (2) and (3), we have
a=103(2a+b)+(4a−3b)=3
and b=14−2a=8. Equation (1) becomes
f(x+y)=f(x)+f(y)+3xy(x+y)+8xy+4.(4)
Putting y=1 in (4), we get
f(x+1)=f(x)+3x(x+1)+8x+5.
For x∈Z+, adding
f(x)f(x−1)f(2)=f(x−1)+3(x−1)x+8(x−1)+5,=f(x−2)+3(x−2)(x−1)+8(x−2)+5,⋮,=f(1)+3(1)(2)+8(1)+5,
we obtain
f(x)=f(1)+3k=1∑x−1k(k+1)+8k=1∑x−1k+5(x−1)=(x−1)x(x+1)+4(x−1)x+5x−4=x3+4x2−4.
This also holds when x=0 as f(0)=−4. Now, putting y=−x in (4), we have
f(0)=f(x)+f(−x)−8x2+4.
For x<0, this yields
f(x)=−f(−x)+8x2−8=−((−x)3+4(−x)2−4)+8x2−8=x3+4x2−4.
So the same formula holds for all x∈Z.
Lastly, for f(x)=x3+4x2−4, we check that
f(x+y)=(x+y)3+4(x+y)2−4=x3+3x2y+3xy2+y3+4x2+8xy+4y2−4
and
f(x)+f(y)+3xy(x+y)+8xy+4=x3+4x2−4+y3+4y2−4+3xy(x+y)+8xy+4.
It is routine to check that the two sides are identical. Thus, this is the only solution.
b.
The greatest possible value of m is −1.
Note that
⇔⇔⇔x3+4x2−4≥mx2+(5m+1)x+4mx3+4x2−x−4≥mx2+5mx+4m(x+1)(x+4)(x−1)≥m(x+1)(x+4)x−1≥m.
By putting x=0, we need m≤−1. When m=−1, this inequality holds for all x≥0. Thus, the greatest possible value of m is −1.