Answer: $f(n) = n+1$, for all positive integers $n$.

Let $f$ be such a solution.

First step. The pre-image of 1 is empty.

Obviously, if there is an integer $x$ so that $f(x) = 1$. Set $a = b = x$ into (1), then
$$f(2x)^2 = f^{f(x)}(x) f^{f(x)}(x) = f(x)^2 = 1$$
This implies $f(2x) = 1$. But, it's impossible since $f$ is injective, as desired.

Second step. $f^{f(n)-1}(n) = 2n$ for all $n \in N$. Moreover, $f(1) = 2$

It's also clear. Set $a = b = n$ into (1) and use the injectivity of $f$:
$$f^{f(n)}(n)^2 = f(2n)^2 \rightarrow f^{f(n)}(n) = f(2n) \rightarrow f^{f(n)-1}(n) = 2n \quad (1)$$
In particular, there is an integer $c$ such that $f(c) = 2$. Then
$$2c = f^{f(c)-1}(c) = f(c) = 2.$$
So $c=1$, as desired.

Third step. The pre-image of 5 is non-empty.

According to the previous work, we get that the pre-images of even integers are non-empty. There must be an integer $d$ such that $f(d) = 4$. By using (1),
$$2d = f^{f(d)-1}(d) = f^3(d) = f^2(4) \qquad (2)$$
Substitute $a = 1, b = 4$ into the problem statement:
$$f^2(4) f^{f(4)}(1) = f(5)^2 \qquad (3)$$
Combining the problem statement with (1), it follows $f(5)$ is an even integer, say $2e$. By (1) again,
$$f(5) = 2e = f^{f(e)-1}(e)$$
Clearly, $e \neq 1$ since $f$ is injective, so $f(e) > 2$ and $5$ is in the image of $f$.

Because the pre-image of 1 is empty, $f^2(g-1)$ and $f^{f(g-1)}(1)$ both are
3. Notice that
$$f^2(g-1) = 3 = f(g) \rightarrow f(g-1) = g$$
Furthermore, take $a = 2, b = g - 2$ in the problem statement
$$f^5(g-2) f^{f(g-2)}(2) = f(g)^2 = 9 \quad (5)$$
By the result of the first step, we then get
$$f^g(1) = f^{f(g-1)}(1) = 3 = f^{f(g-2)}(2) = f^{f(g-2)+1}(1)$$
Therefore, $f(g-2)$ equals to $g-1$. Also, (3) tells us
$$3 = f^5(g-2) = f^2(3).$$
Since
$$f^{f(3)-1}(3) = 6$$
The only possibility is $f(3) = 6$. This means $e = \frac{9}{2}$, which is absurd!

$$f^h (1) = f^{f(h-1)} (1) = 5 = f^{f(h-2)} (2) = f^{f(h-2)} (2) = f^{f(h-2)+1} (1)$$
which implies $f(h-2) = h-1$. And we'll have
$$5 = f^{f(2)} (h-2) = f^{f(2)-3} (5)$$
If $f(2) \neq 3$, then the sequence $5, f(5), f^2(5), \dots$ contains only finite distinct values. On the other hand, (1) tells us that
$$f^{f(5)-1}(5) = 10, f^{f(10)-1}(10) = 20, \dots, f^{f(5 \cdot 2^m)-1}(5 \cdot 2^m) = 5 \cdot 2^{m+1}, \dots$$
In particular, the sequence $5, f(5), f^2(5), \dots$ contains $5 \cdot 2^m$ for all $m \in N \cup \{0\}$. Thus, our assumption is false. That is, $f(2) = 3$ and $4 = f^{f(2)-1}(2) = f(3)$

Sixth step. $f(n) = n+1$ for all $n \in N$

We have shown that the statement $f(n) = n+1$ is true for $n = 1, 2, 3$. Assume that it's true for all $n < k$. If $k$ is odd, write $k = 2l+1$ take $a = l, b = l+2$:
$$f(2l + 2)^2 = f^{f(l)}(l + 2) f^{f(l+2)}(l) = f^{l+1}(l + 2) f^{l+3}(l) = f^{l+1}(l + 2)^2 \\ \Rightarrow 2l + 2 = f^{l+1}(l + 2) = f(2l + 1)$$
For the other case: if $k = 2l$, take $a = l, b = l+1$:
$$f(2l + 1)^2 = f^{f(l)}(l + 1) f^{f(l+1)}(l) = f^{l+1}(l + 1)^2 \\ \Rightarrow 2l + 1 = f^l(l + 1) = f(2l)$$
Thus, use the principle of induction, the conclusion follows.