Solution 1
Square and rearrange the first equation and also rearrange the second. $$\begin{align} a^{2n}-a&=a^2+a+1\\ b^{2n}-b&=3a \end{align}$$ It is trivial that $$\begin{align*} (a-1)^2 > 0 \tag{3} \end{align*}$$ since $a-1$ clearly cannot equal $0$ (Otherwise $a^n=1\neq 1+1$ ). Thus $$\begin{align*} a^2+a+1&>3a \tag{4}\\ a^{2n}-a&>b^{2n}-b \tag{5} \end{align*}$$ where we substituted in equations (1) and (2) to achieve (5). Notice that from $a^{n}=a+1$ we have $a>1$ . Thus, if $b>a$ , then $b^{2n-1}-1>a^{2n-1}-1$ . Since $a>1\Rightarrow a^{2n-1}-1>0$ , multiplying the two inequalities yields $b^{2n}-b>a^{2n}-a$ , a contradiction, so $a> b$ . However, when $n$ equals $0$ or $1$ , the first equation becomes meaningless, so we conclude that for each integer $n\ge 2$ , we always have $a>b$ .
Solution 2
Define $f(x)=x^n-x-1$ and $g(x)=x^{2n}-x-3a$ . By Descarte's Rule of Signs, both polynomials' only positive roots are $a$ and $b$ , respectively. With the Intermediate Value Theorem and the fact that $f(1)=-1$ and $f(2)=2^n-3>0$ , we have $a\in(1,2)$ .
Thus, $-3a\in(-6,-3)$ , which means that $g(1)=-3a<0$ . Also, we find that $g(a)=a^{2n}-4a$ . All that remains to prove is that $g(a)>0$ , or $a^{2n}-4a>0$ . We can then conclude that $b$ is between $1$ and $a$ from the Intermediate Value Theorem. From the first equation given, $a^{2n}=(a+1)^2=a^2+2a+1$ . Subtracting $4a$ gives us $a^2-2a+1>0$ , which is clearly true, as $a\neq1$ . Therefore, we conclude that $1<b<a<2$ .