First, note that the function ι(m)=m is such a function, as
(ι(m))2+2mι(n)+ι(n2)=(m+n)2.
We show that ι is the only such function.
Let f be any such function. Substituting m=n=1, we see that f(1)2+3f(1) must be a square. As (f(1)+1)2≤f(1)2+3f(1)<(f(1)+2)2, the inequality on the left must be an equality, so f(1)=1.
Now let k be any positive integer for which p=2k+1 is an odd prime. By substituting m=k=21(p−1) and n=1, we see that f(k)2+p must be a square, say a2 for a positive integer a. Then p=a2−f(k)2=(a−f(k))(a+f(k)), so by unique factorisation and a−f(k)<a+f(k) it follows that a−f(k)=1 and a+f(k)=p=2k+1. Taking the difference of these two equalities yields f(k)=k.
Now let x be any positive integer, and choose a positive integer k for which p=2k+1 is an odd prime (so that f(k)=k) and that is large enough so that 1−2k−2f(x)<f(x2)−f(x)2<1+2k+2f(x). Substituting m=k and n=x then gives that k2+2kf(x)+f(x2)=(k+f(x))2+f(x2)−f(x)2 must be a square. By our chosen bound on k, we have
(k+f(x)−1)2<(k+f(x))2+f(x2)−f(x)2<(k+f(x)+1)2.
The expression in the middle is a square and therefore must equal (k+f(x))2, therefore f(x2)=f(x)2 for all positive integers x.
Finally, let x be any positive integer, and choose a positive integer k for which p=2k+1 is again an odd prime and that is large enough so that 1−2x−2k<(f(x))2−x2<1+2x+2k. Substituting m=x and n=k then gives that (f(x))2+2xk+k2=(x+k)2+(f(x))2−x2 must be a square. By our chosen bound on k, we have
(x+k−1)2<(x+k)2+(f(x))2−x2<(x+k+1)2.
The expression in the middle is a square and therefore must equal (x+k)2, therefore (f(x))2=x2 for all positive integers x.
Thus f(x)=x for all positive integers x, so f must be equal to ι. Therefore ι is the only function satisfying the required property. ☐