What is the genuine formula for the alignment limit? Is $\sin{(\beta-\alpha)}=-1$ possible also? In the two-Higgs doublet model (2HDM),
the so-called alignment limit is often stated as the one coresponding to $\cos{(\beta-\alpha)}=0$.
Example: Relaxed constraints on the heavy scalar masses in 2HDM (Phys. Rev. D, 2019)
In some other references, it is stated as the one corresponding to $\sin{(\beta-\alpha)}=1$.
example: Search for a charged Higgs boson decaying into a heavy neutral Higgs boson and a W boson in proton-proton collisions at √s=13 TeV (CERN, 2022)
How could that be compatible?
Indeed $\sin{(\beta-\alpha)}=-1$ is another possibility when one has $\cos{(\beta-\alpha)}=0$.
What is the genuine formula for the alignment limit?
 A: It seems like this a matter of taste.  You can see why because $\cos(\beta-\alpha)=0$ and  $\sin(\beta-\alpha)=1$ both for $\beta-\alpha=\pi/2$.  The reason why $\sin(\beta-\alpha)=-1$ is not being considered, is probably because $\beta-\alpha=-\frac{\pi}{2}$ is unphysical or doesn't align with current observational data.  Indeed, in the first paper you linked:

The measurements of the Higgs signal strengths dictate that for type-II 2HDM, at $\tan{β} ∼ 1$, the constraint on $\cos(\beta-\alpha)$ is given by $−0.05 ≲ \cos(\beta-\alpha) ≲ 0.15$ at 95% C.L.

For $−0.05 ≲ \cos(\beta-\alpha) ≲ 0.15$ we have roughly $$81.37≲\beta-\alpha≲92.87 \implies 0.98 ≲ \sin(\beta-\alpha) \le 1,$$ making $\sin(\beta-\alpha)=1$ the only other possibility.
Then again, in the same paper:

This constraint is comparably relaxed in a type-I 2HDM, where the allowed
range is $|\cos(\beta-\alpha)| ≲ 0.4$.

where clearly $\sin(\beta-\alpha)$ has to be positive:
$$87.71≲\beta-\alpha≲ 92.29 \implies 0.99 ≲ \sin(\beta-\alpha) \le 1$$
The two versions of the alignment limit are compatible because they describe the same thing.  So whether you consider  $\sin(\beta-\alpha) \rightarrow 1$ or $\cos(\beta-\alpha) \rightarrow 0$, it doesn't matter.  And as I explained, $\sin(\beta-\alpha) \rightarrow -1$ wouldn't make sense because it describes different physics.
A: According to your second paper,

The two-doublet structure gives rise to five physical Higgs bosons via spontaneous symmetry breaking; two neutral CP-even particles h and H with mh ≤ mH , one neutral CP-odd particle A, and two charged Higgs bosons H±. In these models the ratio of the vacuum expectation values of the two Higgs doublets, tan β, and the mixing angle between h and H, α, are important free parameters. These can be tuned to the alignment limit sin(β − α) = 1 whereby h aligns with the properties of the SM Higgs boson, while the additional Higgs bosons may appear at TeV scale or below it.

I think the way to interpret this paragraph is that the combined phase $\beta-\alpha$ is ”aligned” to make the proposed new particle “orthogonal” to the observed Higgs, which appears to be consistent with the Standard Model.
I don’t know enough about the physics here to know whether the sign of $\beta-\alpha$, whether the new field is “clockwise orthogonal” or “counterclockwise orthogonal,” is a physically meaningful question or not.
Your first reference does discuss the “wrong-sign Yukawa” limit $\sin(\beta+\alpha)=1$ and values for $\tan\beta$, which suggest that the angles $\alpha,\beta$ are constrained separately.
In your first reference, Figure 5 is an “exclusion plot” with $\cos(\beta-\alpha)$ as a parameter.  For this purpose, the $\cos \approx 0$ region is clearly superior to the $\sin \approx 1$ region.  The trig functions are monotonic near their zero crossings, but have rising and falling regions near their maxima or minima.  Consider the transformation
\begin{align}
x & := \frac\pi2 - (\beta-\alpha) & |x| &\lesssim 1
\\
\cos(\beta-\alpha) = \sin x &\approx x - \frac{x^3}{3!} \approx x
\\
\sin(\beta-\alpha) = \cos (-x) &\approx 1 - \frac{x^2}{2!}
\end{align}
If we have reason to believe that some phase $\phi$ is approximately a quarter-turn, looking at $\cos\phi\approx 0$ lets us determine $\phi$ uniquely.  Meanwhile, some value $1 \neq \sin\phi\approx 1$ corresponds to two possible angles.  In addition to the ambiguity, the small-squared dependence on angle also costs precision.  Linear approximations are easier to deal with than quadratic approximations.
A: No idea what the physics is, but, mathematically,
$\cos(\beta-\alpha) = 0$
$\cos^2(\beta-\alpha) = 0$
$1-\sin^2(\beta-\alpha) = 0$
$1= \sin^2(\beta-\alpha) $
$(+ or -)1= \sin(\beta-\alpha)$
Wouldn't they all be equally valid? As are identical.
(Of course by squaring you are allowing the actual value of the difference of angles to be negative as well, but not sure what you want to find out).
