It is clear that we may take g=2 for (a,b)=(1,1). Supposing that (a,b) satisfies the conditions in the problem, let N be a positive integer such that gcd(an+b,bn+a)=g for all n⩾N.
Lemma. We have that g=gcd(a,b) or g=2gcd(a,b).
Proof. Note that both aN+b and aN+1+b are divisible by g. Hence
a(aN+b)−(aN+1+b)=ab−b=a(b−1)
is divisible by g. Analogously, b(a−1) is divisible by g. Their difference a−b is then divisible by g, so g also divides a(b−1)+a(a−b)=a2−a. All powers of a are then congruent modulo g, so a+b≡aN+b≡0(modg). Then 2a=(a+b)+(a−b) and 2b=(a+b)−(a−b) are both divisible by g, so g∣2gcd(a,b). On the other hand, it is clear that gcd(a,b)∣g, thus proving the Lemma.
Let d=gcd(a,b), and write a=dx and b=dy for coprime positive integers x and y. We have that
gcd((dx)n+dy,(dy)n+dx)=dgcd(dn−1xn+y,dn−1yn+x),
so the Lemma tells us that
gcd(dn−1xn+y,dn−1yn+x)⩽2
for all n⩾N. Defining K=d2xy+1, note that K is coprime to each of d,x, and y. By Euler's theorem, for n≡−1(modφ(K)) we have that
dn−1xn+y≡d−2x−1+y≡d−2x−1(1+d2xy)≡0(modK),
so K∣dn−1xn+y. Analogously, we have that K∣dn−1yn+x. Taking such an n which also satisfies n⩾N gives us that
K∣gcd(dn−1xn+y,dn−1yn+x)⩽2
This is only possible when d=x=y=1, which yields the only solution (a,b)=(1,1).
For any prime factor p of ab+1,p is coprime to a and b. Take an n⩾N such that n≡−1(modp−1). By Fermat's little theorem, we have that
an+b≡a−1+b=a−1(1+ab)≡0(modp),bn+a≡b−1+a=b−1(1+ab)≡0(modp),
then p divides g. By the Lemma, we have that p∣2gcd(a,b), and thus p=2. Therefore, ab+1 is a power of 2, and a and b are both odd numbers.
If (a,b)=(1,1), then ab+1 is divisible by 4, hence {a,b}={−1,1}(mod4). For odd n⩾N, we have that
an+b≡bn+a≡(−1)+1=0(mod4)
then 4∣g. But by the Lemma, we have that ν2(g)⩽ν2(2gcd(a,b))=1, which is a contradiction. So the only solution to the problem is (a,b)=(1,1).