Since a,b∈(0,2π) we have sina,sinb,cosa,cosb>0.
Then
sinbsin3a+cosbcos3a=sinasinbsin4a+cosacosbcos4a=sinasinb(sin2a)2+cosacosb(cos2a)2≥cosacosb+sinasinb(sin2a+cos2a)2=cos(a−b)1
where we have used the Cauchy-Schwarz inequality. We have equality if and only if
sinasinbsin2a=cosacosbcos2a
that is sinacosb−sinbcosa=0, hence sin(a−b)=0 yielding a=b.