The number of ways to grow a forest that may or may not be balanced is equal to $n^2!$. To answer the question we will first count the unbalanced forests, then subtract.

The main observation is that an unbalanced forest contains exactly one tree that violates both conditions. Indeed, if there were two such trees, say of heights $i$ and $j$, then they cannot be in the same row or same column since $i \neq j$. If tree $i$ is in square $(a, b)$, then $i$ is the tallest tree in row $a$ and the smallest tree in column $b$. Similarly, if tree $j$ is at position $(m, n)$, then $j$ is the tallest tree in row $m$ and the smallest tree in column $n$. Therefore, if the tree in square $(a, n)$ has size $u$, then $j < u < i$, hence $j < i$. If the tree in square $(m, b)$ has size $v$, then $i < v < j$, a contradiction.

To plant an unbalanced forest, there are $n^2$ possible choices for the square with the tree that violates both conditions. The row and column that contain this square will together accommodate $2n-1$ trees. There are $\binom{n^2}{2n-1}$ possible choices for these trees. Once we have chosen $2n-1$ numbers, the middle value (with $n-1$ smaller and $n-1$ larger numbers in the chosen set) has to become the tree that violates both conditions. The $n-1$ smaller trees can be planted in $(n-1)!$ ways in their row and similarly for the larger trees.

The remaining $n^2 - (2n-1) = (n-1)^2$ trees can be planted as we like in the remaining squares of the grid. We have $(n-1)^2!$ possibilities for that. In total we find that there are
$$n^2 \cdot \binom{n^2}{2n-1} \cdot ((n-1)!)^2 \cdot (n-1)^2!$$
ways to plant an unbalanced forest, hence the number of ways to plant a balanced forest is equal to
$$n^2! - n^2 \cdot \binom{n^2}{2n-1} \cdot ((n-1)!)^2 \cdot (n-1)^2!$$