For a prime p and a positive integer n, denote by ordpn the largest nonnegative integer i such that pi divides n.
First we consider the case y≡0(mod2). Since 3z=8y+2xy+1≡1(mod4), x is even. Write x=2z and y=2w for positive integers z and w, and we have 8zw+1=32z−82w=(3z−8w)(3z+8w)≥3z+8w. 3z−8w>0 shows z>w, hence we have 8z2+1>8zw+1>3z. If z≥6, by the binomial theorem we have
3z=(2+1)z≥2z−2⋅zC2+2z−1z+2z>16⋅zC2+32z+1>8z2+1,
which is absurd. If z=5, 3z>8z2+1 holds. It follows that z=1,2,3,4 is necessary. Since 94≥9z>64w, w=1,2 is necessary. Moreover, 3z+8w≤8zw+1≤65 shows that w=1 is necessary. Among z=1,2,3,4, only z=2 satisfies 9z=8z+65. Therefore the solution in this case is (z,w)=(2,1), that is, (x,y)=(4,2).
Next we consider the case y≡1(mod2). From 8y+1=3z−2xy we have ord3(8y+1)=ord3(3z−2xy). Here we use the following lemma:
Lemma. For any positive odd integer y, ord3(8y+1)=ord3y+2.
Proof. We proceed by induction on y. If y=1, ord3(8y+1)=ord3y+2=2, thus the claim is true. Let ℓ≥3 be an odd integer and assume that the claim is true for y<ℓ.
If ℓ is not a multiple of 3, we can write ℓ=3k+r with a nonnegative integer k and r∈{1,2}. Then we have 8ℓ=8r×512k≡(−1)k8r(mod27). Since r∈{1,2}, 8ℓ+1≡0(mod27). On the other hand, since ℓ is odd we have 8ℓ+1≡0(mod9), thus ord3(8ℓ+1)=2. Therefore the claim is true when ℓ is not a multiple of 3.
If ℓ is a multiple of 3, we can write ℓ=3k for a positive odd integer k. Then we have ord3(8ℓ+1)=ord3(8k+1)+ord3(64k−8k+1), for 8ℓ+1=(8k+1)(64k−8k+1). Since k<ℓ, inductive hypothesis shows that ord3(8k+1)=ord3k+2. From 64k−8k+1≡3(mod9) we have ord3(64k−8k+1)=1. It follows that ord3(8ℓ+1)=ord3k+3=ord3ℓ+2. Therefore the claim is also true when ℓ is a multiple of 3 and the induction is complete. ■
Since 0<3z−2xy<3z we have ord3(3z−2xy)<x. This and the lemma show that ord32xy=ord3(3z−2xy)=ord3y+2, thus we have ord3x=2. Hence x is a multiple of 9 and can be written as x=3v where v is a multiple of 3. Then we have 6vy+1=33v−23v=(3v−2v)(9v+3v⋅2v+4v)>9v. From 3v−2v>0, we have 2v>y and 12v2+1>6vy+1>9v holds. Since v≥3, the binomial theorem shows that
9v=(8+1)v≥64⋅vC2+8v+1>12v2+1,
which is a contradiction. As a result, there exist no solutions when y is odd.
From above, the answer is (x,y)=(4,2).