There are angles A,B, and C in the interval (0∘,180∘) such that
x=tan2A,y=tan2B,z=tan2C.
By the addition and subtraction formulas, the second equation in the given system becomes
1=tan2Atan2B+tan2Btan2C+tan2Ctan2A=tan2Atan2B+tan2Ctan2A+B(1−tan2Atan2B),
(1−tan2Ctan2A+B)(1−tan2Atan2B)=0.
Note that tan2Atan2B=xy=1, otherwise z(x+y)=0, implying z=0, which is impossible. Therefore, tan2Ctan2A+B=1, or 2A+B+2C=90∘. In other words, A,B,C are the angles of a triangle. Let ABC denote that triangle.
We rewrite the first equation in the given system as
5(x2+1)x=12(y2+1)y=13(z2+1)z.
Note that, by the double-angle formula, we have
x2+1x=tan22A+1tan2A=sec22Atan2A=sin2Acos2A=2sinA,
and analogously for the expressions of y and z. We conclude that
5sinA=12sinB=13sinC.
By the sine rule, the sides of triangle ABC are in ratio 5:12:13 with sinA=135, sinB=1312 and sinC=1. Hence x2+1x=265, or 5x2−26x+5=0, implying that x=5 or x=51. Likewise, we have y=23 or y=32, and z=1. Substituting z=1 into the second equation in the given system leads to xy+x+y=1, implying that (x,y,z)=(51,32,1) is the only solution with x>0. Hence (51,32,1) and (−51,−32,−1) are the solutions of the problem.