Let ABQ1P be a rhombus and let segments AQ1 and BP intersect at Q2. Let X be a point on ray Q2Q1. Because lines PQ2 and Q1Q2 are perpendicular, PX increases as X moves away from Q2. Note that AQ1 bisects angle BAP, that is, Q is on the ray Q2Q1. Because ∠ABC>∠BCA, A and B are on the same side of line PQ. Because PQ1∥AB and PQ⊥BC, ∠CBA is obtuse if and only if the segment PQ1 and B are on different sides of line PQ. It follows that ∠CBA is obtuse if and only if Q1 is farther from Q2 on ray Q2Q1 than Q, that is, PQ<PQ1=AB.
Second Solution. (by Reid Barton) We may assume without loss of generality that P is closer to A than is Q. Let X be the intersection of segment AC and ray QP, and let A1 be the reflection of A across the line PQ. By symmetry, A1 lies on the ray BX and AA1∥BC.
Let ∠BCA=x. Then ∠A1AC=x=∠XBC. Since ∠ABC=2x, ∠ABA1=x. It follows that
∠PXA=∠PXB+∠BXA=90∘−∠XBC+∠BXA=90∘+∠BCX=90∘+x.
Since ∠CAB=180∘−3x, it follows that ∠XAP=∠CAP=60∘−x. Hence,
∠APX=180∘−∠PXA−∠XAP=30∘.
Hence, ∠APA1=2∠APX=60∘. Because A1P=AP, this implies that triangle PAA1 is equilateral and that AA1=AP. Note also that ∠BA1A=∠A1BC=∠ABA1. Hence, AB=AA1=AP. Therefore, PQ<AB if and only if PQ<AP. In triangle APQ, PQ<AP if and only if ∠PAQ<∠AQP, that is,
∠PAQ<∠APX−∠PAQ=30∘−∠PAQ,
or 2∠PAQ<30∘. Because ∠PAQ=60∘−x, we conclude that PQ<AB if and only if
90∘<2x=∠ABC,
as desired.
Third Solution.
As in the first two solutions, let ∠BCA=x and assume that ∠CAB=3∠CAP. Also, let ∠PBC=y. Then ∠CAB=180∘−3x, ∠PAB=120∘−2x, ∠ABP=2x−y, and ∠BPA=60∘+y. From the Law of Sines in triangles ABC, ABP, and BCP, we obtain
sin(180∘−3x)BCsin(60∘+y)ABsinyPC=sinxAB,=sin(120∘−2x)BP,=sin(180∘−2y)BC.
Multiplying the above relations and taking into account that BP=PC yields
sin3xsin(60∘+y)siny=sinxsin(60∘+2x)sin2y,
or
sinxsin3xsin(60∘+y)=sinysin2ysin(60∘+2x).
By the Triple-angle formulas, sin3x=sinx(3−4sin2x). By the Double-angle formulas, sin2y=siny⋅2cosy. It follows that
(3−4sin2x)sin(60∘+y)=2cosysin(60∘+2x),
or, by the Double-angle formulas,
(1+2cos2x)sin(60∘+y)=2cosysin(60∘+2x).
Applying the Product-to-Sum formulas gives
sin(60∘+y)+sin(60∘+y+2x)+sin(60∘+y−2x)=sin(60∘+2x+y)+sin(60∘+2x−y).
We obtain
sin(60∘+y)=sin[60∘+(2x−y)]−sin[60∘−(2x−y)],
or, using the Difference-to-Product formulas,
sin(60∘+y)=2cos60∘sin(2x−y)=sin(2x−y).
Note that 60∘+y>0, 2x−y>0, and that (60∘+y)+(2x−y)=60∘+2x<180∘ (as 3x=180∘−∠BAC<180∘). It follows that 60∘+y=2x−y, i.e., ∠BPA=∠ABP.
Hence, triangle ABP is isosceles and AP=AB. It follows that PQ<AB is equivalent to PQ<AP. In triangle APQ, this is equivalent to
60∘−x=∠PAQ<∠AQP=x−30∘,
that is, ∠B=2x>90∘. It follows that PQ>AB if and only if ∠B is obtuse.