How is it possible to take the inner product of states which belong to two different Hilbert spaces? Question 1 In case of spontaneous breakdown of a continuous symmetry e.g. the ${\rm U(1)}$ symmetry, two different vacua can be labelled as $|\theta\rangle$ and $|\theta^\prime\rangle$, and they belong to different Hilbert spaces. I have often encountered inner products of the form $$\langle\theta|\theta^\prime\rangle=\delta(\theta-\theta^\prime)$$ for example, in the question here by @innisfree. Is it mathematically meaningful to take the inner product of two states which belong to two different Hilbert spaces?
Question 2 One must also look at the Wikipedia article here, which gives an expression of the form $\langle 0|J_0(0)|\theta\rangle\neq0$ where $J_0$ is the zeroth component of the current $J_\mu$ corresponding to the spontaneously broken symmetry.
$\bullet$ Which Hilbert space does $J_0$ act on? Does it act on the Hilbert space containing the vacuum $|0\rangle$ or that containing the vacuum $|\theta\rangle$? 
$\bullet$ Which Hilbert space does the state $J_0|0\rangle$ or $J_0|\theta\rangle$ live? Does it belong to the Hilbert space containing the vacuum $|0\rangle$ or that containing the vacuum $|\theta\rangle$?
$\bullet$ In which Hilbert space is this scalar product $\langle 0|J_0(0)|\theta\rangle$ defined?
 A: Regarding question 1:
Two Hilbert spaces ${\cal H}_A$ and ${\cal H}_B$ can always be regarded as mutually orthogonal subspaces of a single Hilbert space ${\cal H}$. The algebra of observables might not contain any operators that connect these two subspaces, but I don't think there's any problem considering relationships like $\langle a|b\rangle=0$ with $|a\rangle\in{\cal H}_A$ and $|b\rangle\in{\cal H}_B$.
However, since the question refers to a spontaneously broken continuous symmetry, with the family ground states parameterized by a continuous parameter $\theta$, there is a problem with the equation $\langle \theta|\theta'\rangle=\delta(\theta-\theta')$. The problem is that this equation would imply the existence of an uncountably infinite number of mutually orthogonal state-vectors, which would imply that the Hilbert space is not separable. Quantum (field) theory is normally based on a separable Hilbert space. So in this case, I think the answer is that these different vacuum states cannot all belong to the same (separable) Hilbert space, so equations involving their inner products are dubious.
Granted, even introductory QM texts often write things like $\langle x|x'\rangle=\delta(x-x')$ in reference to the "eigenstates of the position operator", but that's also ill-defined for the same reason. The position operator does not have eigenstates, and it's not defined on the whole Hilbert space anyway because it's unbounded.
Regarding question 2:
I don't see how a local operator can connect two different SSB ground states. However, we can have a relationship of the form $\langle 0|J_0(x)|\pi\rangle\neq 0$, where $|0\rangle$ is a ground state and $|\pi\rangle$ is a state containing a single Goldstone boson, where that single boson lives in the world whose ground state is $|0\rangle$, not in a world whose ground state is $|\theta\rangle$ with $\theta\neq 0$. This relationship is shown, for example, on page 332 in this paper: "Spontaneous Breakdown of Symmetries and Zero-Mass States", https://projecteuclid.org/euclid.cmp/1103840121, accessed on 2018-10-26. The notation in the Wikipedia article is not very clear about this, but I suspect that this is what it meant.
To corroborate this guess about what the Wikipedia article might have meant, the same relationship (a local current connecting a vacuum state to a one-boson state, not another vacuum state) is shown in Weinberg, The Quantum Theory of Fields, Volume II, equation (19.2.34), in precisely the same context of a spontaneously broken global symmetry.
