Let p be the least prime factor of b, and q be the least natural number such that p∣(aq−1) (such a number exists because p∣(ab−1)). From Little Fermat's theorem, we have that p∣(ap−1−1), which implies q∣b and q∣(p−1), and from the minimality of p, we get that q=1, i.e. p∣(a−1). Let b=pαc, where c is not divisible by p. If p2∣(a−1), then a≡1(modp2), so ak≡1(modp2), for every natural number k from (al(p−1)+al(p−2)+⋯+1)≡p(modp2) and (apα(c−1)+apα(c−2)+⋯+1)≡c(modp2), apαc−1=(a−1)(ap−1+ap−2+⋯+1)…
…(apα−1(p−1)+apα−1(p−2)+⋯+1)(apα(c−1)+apα(c−2)+⋯+1)
so the degree of p in a−1ab−1 is α and therefore we get that pα(a−1)∣(a−1)
(pαa∣(ab−1)). The last is not possible because for α≥1, p≥2 and a≥2, it holds that pα(a−1)>a−1.
This implies that, the degree of p in a−1 must be 1. If p>2, we will prove that the degree of p in apk−1 is k+1, for every natural number k. For k=0, the statement is true. Let the statement be true for every k<n. For k=n from the following equalities
apn−1=(apn−1)p−1=(apn−1−1)(apn−1(p−1)+apn−1(p−2)+⋯+1)
and
alpn−1−1=(apn−1−1)(apn−1(l−1)+apn−1(l−2)+⋯+1)≡pndnl(modpn+1)
we get that
apn−1≡pndn(pndn(p−1+p−2+⋯+1)+p)≡≡pn+1dn(pndn2p−1+1)(modp2n+1)
(where dn=pnapn−1−1 and dn and p coprime by the assumption) with which we've proved the statement by induction. From the equality
apαc−1=(apα−1)(apα(c−1)+apα(c−2)+⋯+1)
and from apα(c−1)+apα(c−2)+⋯+1≡c(modp), we get that the degree of p in ab−1 is α+1, and on the other hand it must be greater or equal to αa, which is possible only for α=1 and a=2, but this case is not possible because of p∤(a−1).
Now, the case when p=2 remains. From the equality (the same as above for p>2)
a2αc−1=(a−1)(a+1)…(a2α−1+1)(a2α(c−1)+a2α(c−2)+⋯+1).
The fact that c is odd implies that a2α(c−1)+a2α(c−2)+⋯+1 is an odd number as a sum of an odd number of odd numbers (a has to be odd, because 2∣(a−1)). From 2∣(a−1) and 2∣(a+1), we get that 4∣(a2k−1), for every natural number k, but this implies that 4 is not a divisor of a2k+1. This implies that the degree of 2 in a2−1ab−1 is α−1, so because 2αa∣(ab−1), we get that 2α(a−1)+1∣(a2−1), and because 4 can be a divisor of only one of a+1 and a−1, we get that 2α(a−1)≤a+1, which is possible if and only if α=1 or a=3. Let r be the least prime factor of c, c=rβd, r and d are coprime and s is the least natural number for which r∣(3s−1). This implies that s∣b and s∣(r−1) (the same as before for p). This is possible only if s=1 or s=2 (those are the only divisors of b, less than r). In both cases r∣(32−1), which is possible only for r=2, a contradiction with the choice of r. We get that the unique solution is b=2 and a=3 (ba=8=ab−1).