The potential difference between two points is minus the work done by the electric field on unit positive charge in going from one point to the other.
Let the distance between $A$ and $B$ be $L$ as shown in the diagram.
$\vec E = -E\, \hat y$ and $d\vec l = dy \, \hat y$ where $-E$ and $dy$ are components of the electric field and displacement in the $\hat y$ direction.
Minus the work done by the electric field after undergoing a displacement $d\vec l$ is
$-(\vec E \cdot d\vec l) =-(-E\,\hat y \cdot dy\, \hat y) = + E\,dy$
Note that no choice has been made about the direction of the total displacement.
Going from $B$ to $A$
$\displaystyle V_{\rm A} - V_{\rm B} = - \int ^A _B \vec E \cdot d\vec l = \int ^L _0 E\,dy = +EL$
Going from $A$ to $B$
$\displaystyle V_{\rm B} - V_{\rm A} = - \int ^B _A \vec E \cdot d\vec l = \int ^0 _L E\,dy = -EL$
Now perhaps the integrations are masking what is happening so because the electric field is constant I can write that minus the work done by the electric field $\vec E$ in undergoing a displacement of $\Delta \vec l$ from $B$ to $A$ is $- \vec E \cdot \Delta \vec l$.
$\vec E = - E \, \hat y$ as shown in the diagram and $\Delta \vec l = +L\,\hat y$ when going from $B$ to $A$.
and so minus the work done by the electric field is $-(- E \, \hat y \cdot (+L\,\hat y)) = +EL$
Going the other way from $A$ to $B$ the electric field stays the same but now the displacement $\Delta \vec l = -L \, \hat y$ and so minus the work done by the electric field is $-EL$.
So when you integrate the limits of the integration give you the direction of travel and in effect decide whether the incremental steps are positive (in the $\hat y$ direction) or negative (in the $-\hat y$ direction).
With your introduction of $\cos \phi$ you were in effect choosing the direction of travel for the test charge and then in one of you integrals interposing a different direction of travel by your choice of limits of integration.