a. First, we prove by induction that 0<xn<2n3, ∀n≥1.
The base case n=1 is trivial. For n=2, we have
0<x12(3−2x1)<3x12<43⇒0<x2<43.
Assume that 0<xk<2k3 for some k≥2, by the AM-GM inequality,
0<xk2(3−2kxk)=k21(kxk)(kxk)(3−2kxk)≤k21<2(k+1)3.
Therefore, 0<xk+1<2(k+1)3. The inductive step is completed.
By the Squeeze theorem, we have limn→∞xn=0.
b. It is clear that xn+1=3xn2−2nxn3≤3xn2 implies
xn+2≤3xn+12≤27xn4≤16n437=n4C,∀n≥1.
For all n>2, we obtain that
yn=i=1∑nixi=x1+2x2+i=1∑n−2(i+2)xi+2≤x1+2x2+Ci=1∑n−2i4i+2≤x1+2x2+3Ci=1∑n−2i21.
Hence, (yn) is bounded above. Note that (yn) is increasing, we conclude that (yn) is convergent. □