In this Review, section 3.1 page 58, there is a paragraph that answers your question:
"A $U(N)$ gauge theory is essentially equivalent to a free $U(1)$ vector multiplet times
an $SU(N)$ gauge theory, up to some $Z_N$ identifications (which affect only global issues). In the dual string theory all modes interact with gravity, so there are no decoupled
modes. Therefore, the bulk $AdS$ theory is describing the $SU(N)$ part of the gauge
theory. In fact we were not precise when we said that there were two sets of excitations
at low energies, the excitations in the asymptotic flat space and the excitations in
the near horizon region. There are also some zero modes which live in the region
connecting the “throat” (the near horizon region) with the bulk, which correspond to
the $U(1)$ degrees of freedom mentioned above. The $U(1)$ vector supermultiplet includes
six scalars which are related to the center of mass motion of all the branes [151]. From
the $AdS$ point of view these zero modes live at the boundary, and it looks like we might
or might not decide to include them in the $AdS$ theory. Depending on this choice we
could have a correspondence to an $SU(N)$ or a $U(N)$ theory. The $U(1)$ center of mass
degree of freedom is related to the topological theory of $B$-fields on $AdS$ [152]; if one
imposes local boundary conditions for these $B$-fields at the boundary of $AdS$ one finds
a $U(1)$ gauge field living at the boundary [153], as is familiar in Chern-Simons theories
[25, 154]. These modes living at the boundary are sometimes called singletons (or
doubletons)"
As you can see it is up to you to include the $U(1)$ part in the $SYM$ side of the duality. This choice is related to the choice of including or not local boundary conditions to the $B$-field in the $AdS$ side of the duality. See also the reference inside this Review, like this one