We factorise both equations and obtain
x(x+1)=(y−1)y(y+1),(1)
y(y+1)=(x−1)x(x+1).(2)
We insert (2) into (1) and obtain
x(x+1)=(y−1)(x−1)x(x+1).(3)
We first consider the cases x=0 and x=−1, where (2) results in y∈{0,−1}. Thus we obtained the solutions
(0,0),(0,−1),(−1,0),(−1,−1).
From now on, we assume that x∈/{0,−1}. Cancelling x(x+1) in (3) yields
1=(y−1)(x−1).(4)
In particular, we have x=1 and y=1. This is equivalent to
y=1+x−11=x−1x⟺y+1=x−12x−1.
We insert this into (2) and obtain
x−1x⋅x−12x−1=x(x−1)(x+1)
which is equivalent to
2x−1=(x−1)3(x+1)⟺2x−1=(x2−2x+1)(x2−1)⟺2x−1=x4−2x3+x2−x2+2x−1⟺x4−2x3=0⟺x=2.
because x=0. Inserting this in (4) yields y=2, so we got the fifth solution (2,2). Therefore, all solutions are given as
(0,0),(0,−1),(−1,0),(−1,−1),(2,2).