Physical relevance of $\varphi \in \operatorname{Aut}(G)$ in Fourier transform on $L^2(G)$ Let $S$ be a quantum mechanical system described by a Hilbert space $\mathbb{H}_S=L^2(G)$, where $G$ is a locally compact abelian group (think of $G=\mathbb{R^n}$ or $G$ finite). Let $\widehat{G}=\operatorname{Hom}(G,S^1)$ be the Pontryagin dual of $G$.
The Fourier transform $\mathcal{F}:L^2(G) \to L^2(\widehat{G})$ is often used to switch back and forth between two orthonormal bases in $\mathbb{H}_S$, for example the eigenvectors of position and momentum operators for $G=\mathbb{R^3}$. For this two work, we actually need an isometry $\mathcal{F}':\mathbb{H}_S \to \mathbb{H}_S$ which can be achieved by fixing a (continuous? measurable?) isomorphism $\psi:G \to \widehat{G}$, so that $\mathcal{F'}=\psi^*\circ \mathcal{F}$. However, there are two issues here: first, in general $G$ and $\widehat{G}$ don't have to be isomorphic and second, if it exists $\psi$ is not canonical.
A solution to both issues would be to drop the assumption that position and momentum are actually measured in the same Hilbert space but in the respective duals. Maybe everyone knows this, but so far I thought that there is only one Hilbert space for every system.
A solution to the second issue would be to consider a whole family of "Fourier transforms" $\mathcal{F}'_\varphi$ parameterized by $\varphi \in \operatorname{Aut}(G)$. In this case, the question is what physical relevance this parameter $\varphi$ has.
The latter (fixing a $\psi$) is what I usually see. For example in quantum computing people define the quantum Fourier transformation on $G=\mathbb{Z}/N\mathbb{Z}$ by fixing a primitive $N$th root of unity $\omega_N$, which is usually chosen to be $\omega_N=e^{\frac{2 \pi i}{N}}$ but could also be $\omega_N^k$ for $\gcd(k,N)=1$, $0 \leq k \leq N-1$.

Disclaimer: The next two paragraphs are nonsense. I was under the impression that we need to switch to position/momentum space in order to measure position/momentum, which is not the case.
In the accepted answer it seems that the system is described by at least two Hilbert spaces, $L^2(G)$ and $L^2(\widehat{G})$. You might argue that these are isometric, so physically it's actually just one Hilbert space. However, I'm not happy with this argumentation. Sure, they are isometric, but the isometry is the Fourier transform which contains important information. So I would rather think of the system as a groupoid (i.e. a category with only isomorphisms) of Hilbert spaces connected by unitary operators. The benefit here is, that by having different objects we can keep track of what type of observable we are measuring. If you think this is not really the right picture, please let me know.

Thinking a bit about this, I realized that for the harmonic oscillator $$\hat{H}=\frac{\hat{p}^2}{2m}+\frac{m \omega^2 \hat{x}^2}{2}=\hbar \omega \left(\hat{a}^\dagger \hat{a} + \frac{1}{2} \right)$$ we measure both $\hat{x}$ and $\hat{p}$ at the same time, so (at least at first sight) there seems to be something wrong in the above picture.

Let $G$ be a finite abelian group. Then $L^2(G)=F(G)$ is just the space of functions from $G$ to $\mathbb{C}$. We set
$$\langle f | g \rangle = \sum_{x \in G} f(x) \overline{g(x)} \qquad f,g \in F(G)$$
Then we already have two canonical orthonormal bases present in $F(G)$: First, the set of functions $\{ \delta_x\}$, where $\delta_x(y)=\delta_{x,y}$ and second, the set of normalized characters $\{ f_\chi=|G|^{-\frac{1}{2}}\chi\}$, where $\chi \in \widehat{G}$. I would like to think of the $\delta_x$'s as momentum eigenvectors and of the $f_\chi$'s as position eigenvectors in momentum space. When we apply the Fourier transform, we obtain
$$\mathcal{F}(\delta_x)=f_{x^{-1}} \qquad \mathcal{F}(f_\chi)=\delta_{\chi}$$ in the position space $F(\widehat{G})$, where we identified $G$ with it's double dual in the first equation. So we can measure both observables in both spaces and the Fourier transform just identifies two fully self-sufficient spaces. I think this was the missing link for me, sorry for making it more confusing then necessary..
 A: The Fourier transform itself is what allows you to identify the two Hilbert spaces $L^2(G)$ and $L^2(\hat G)$. They are canonically isomorphic, even if the underlying groups are not (which is the case, most of the time). There is no need to do anything more complicated.
If you pick your favorite orthonormal basis in $L^2(\hat G)$ (typically, the "position" basis), you can pull it back to $L^2(G)$ using $\mathcal F$ and obtain the actual momentum eigenbasis.
For an explicit example, consider the $n$-dimensional torus  $G = \mathbb T^n = \mathbb R^n / \mathbb Z^n$, whose Pontryagin dual is $\hat G = \mathbb Z^n$.
The "position eigenbasis" of $L^2(\hat G)$ contains the kets $|k\rangle$ localized at a single site $k\in \mathbb Z^n$.
The momentum eigenstates are plane waves $|\psi_k\rangle \in L^2(G)$ whose wave function is $\psi_k(x) = \exp(2i\pi x\cdot k)$ for some $k\in\mathbb Z^n$ is sent by the Fourier transform to :
$$\mathcal F|\psi_k\rangle = |k\rangle$$
In other words, if we know the basis $|k\rangle$ in $L^2(\hat G)$, we can find the momentum eigenstates by applying $\mathcal F^{-1}$.
