For real numbers $r$ and $s$ we call them equivalent (and write $r \equiv s$), if $r - s \in \mathbb{Z}$. Then,
$$g(x) \equiv \begin{cases} x - a, & \text{if } x \not\equiv a, b, \\ b - a, & \text{if } x \equiv a, \\ -a, & \text{if } x \equiv b. \end{cases}$$
Note that $g$ is a bijection on $(0,1)$. Let
$$S = \{g^{-i}(a) : 0 \le i \le n-1\} \cup \{g^{-i}(b) : 0 \le i \le n-1\}.$$
Then $g^n(x) = x - na$ for $x \notin S$.
Case 1. $g^n$ has a fixed point $x'$ outside $S$.
Then $x' \equiv g^n(x') \equiv x' - na$, and therefore, $na \equiv 0$ and $g^n(x) = x$ for all $x \notin S$. On the other hand, $g^n$ is a permutation of $S$, and for some positive integer $m$, $(g^n)^m$ be the identity in $S$. So we can take $N = mn$ in this case.
Case 2. All fixed points of $g^n$ are in $S$.
If $g(a) \notin S$, then none of $g^{-i}(a)$ for $0 \le i \le n-1$ can be a fixed point of $g^n$, and $g^n$ can not have more than $n$ fixed points. Similarly, $g(b) \notin S$ is impossible. Therefore, $g(S) = S$. In particular, as in Case 1, there is a positive integer $m$ such that $g^{nm}(x) = x$ for all $x \in S$.
Since $g(S) = S$, $g^n$ is a bijection on $(0,1)\setminus S$. As $g^n$ is defined by $x \to x - na$ on $(0,1)\setminus S$, it means that $x \to x - na$ is also bijection on $S$. It follows that $lna \equiv 0$ for some positive integer $l$, and therefore $g^{nl}(x) = x$ for all $x \notin S$.
Hence we can choose $N = nml$ in this case.