On state transformations and the requirement of space-time invariance in (non-relativistic) quantum mechanics I am trying to follow the development in Ballentine's Quantum Mechanics: A Modern Development but am struggling a lot. Please excuse my attaching of a picture of the development, but my question quite literally has to do with not being able to parse things so I fear it's a crucial inclusion. I also think that, though this question references a specific book, the crux of this question is widely applicable: what does it mean to say that the laws of physics are invariant under certain transformations, and how does that relate to corresponding transformations of the quantum mechanical Hilbert space.
My question:
With respect to page 63, bullet point (a): why must the eigenvalue $a_n$ NOT change in the transformed system? If I have $Q|x\rangle = x|x\rangle$ and I transform the system via a displacement (active point of view, but I suppose in the passive point of view I think of things as looking at the same system from a different frame of reference) then I want the $|x'\rangle$ corresponding to $|x\rangle$ to obey $Q|x'\rangle = x'|x'\rangle$, don't I? I think I have a fundamental misunderstanding of what "invariant under certain space-time symmetry operations" ought to mean.
Edit: I have found this answer which, unfortunately, is not entirely satisfactory. In particular, Ballentine seems to say that the properties (1) and (2) given are what we must demand because they are fundamental to how spacetime transformations "work", and then goes on to show in his equation (3.2) that this implies the transformation rule for operators $A' = UAU^\dagger$ for some unitary or anti-unitary operator $U$. Thus presupposing that transformations of operators ought to be of this form is begging the question.
Edit 2: I have added more pictures as context for the discussion. As I explain in more detail below, I think the crux of things here is my misunderstanding of what the "priming" operation which Ballentine does in defining the physical/experimental requirements (a) and (b) means. Feel free to ignore the extended discussion beforehand. The last paragraph is what matters.
Note this is very long, and it represents my understanding of Ballentine's entire Chapter 3.1; again, skip to the next/last paragraph for the question proper. We begin by noting the physical assumption/experimental fact that the measurements we do on any system must be/are invariant under certain transformations (in the nonrelativistic limit, these will be the Galilean transformations; although how we actually think about doing those transformations is unclear to me as per the last paragraph to follow), with invariance here meaning that these transformations (at this point represented by some unclear priming (') operation) are such that (a) and (b) on p.63 are respected. We then invoke Wigner's theorem, which says that any transformation obeying (b) can be written as a unitary or antiunitary transformation. We further note (again, a physical fact) that the transformations under which our universe displays invariance are all continuous, so that the argument given by Ballentine means that in our universe, all of the transformations under which (a) and (b) are respected are unitary. Now the preceding development, applied to our requirement (a), gives us the transformation rule (3.2) for operators! After equation (3.2), we then develop Stone's theorem formally (in the sense of not rigorously); every continuous family of unitary operators $U(s)$ (with respect to one parameter) is generated by some self-adjoint operator $K$ -- this is equation (3.4). That is, every transformation of a system which demands invariance in the sense of (a) and (b) are generated by some $K$. In Chapter 3.2, we'll make precise exactly what this set of transformations is. Here, we've just established that we can write them as in (3.4).
The only thing I struggle to make rigorous is what exactly is meant by "transformation" above. I think the crux of this is what Ballentine means by priming (') in (a) and (b). Are we saying that we imagine (actively) transforming both the state and our coordinate system somewhere else in space/time (e.g. we shift our state and our coordinate system by some displacement, or we rotate our state and our coordinate system by some angle) and use that the laws of physics (i.e. any measurement we could make of any observable $A$) is identical in that case too? Is there any way to make this all precise?
Pictures:




 A: This really isn't anything specific to quantum mechanics, this is a general statement about how passive coordinate transformations work in physics:
Consider a classical observable $f$ that yields some value $f(s)$ for some state $s$ (this is an abstract point in phase space, not some tuple of coordinates). In one choice of phase space coordinates $(x,p)$, we find that $f(s) = f(x,p) = x$, i.e. it is just position. Now, in the coordinate system with $x' = x + a$ for some shift constant $a$ (i.e. the first system translated by $a$), we then have that $f(s) = x = x' - a$, so $f(x',p') = x' - a$, since changing coordinates on phase space cannot change the value that $f$ has on the state $s$.
The eigenvalue of the operators $A$ and $A'$ in your excerpt plays the role of the $f(s)$, and the $A$ and $A'$ are the analogue of the different coordinate expressions $f(x,p)$ and $f(x',p')$, while the $\lvert\phi\rangle$ and $\lvert\phi'\rangle$ are the analogues of the different coordinate expressions $(x,p)$ and $(x',p')$ of a state (i.e. classically a point in phase space).
That is, when $A$ as in your example is a position operator, $A$ represents "measure distance with respect to a certain origin O" and $A'$ still represents "measure distance with respect to O as the origin" - the transformation isn't changing what the observable means, it's just changing your expression for the observable as a concrete operator because you changed how you describe the point O in your maths, so $A$ and $A'$ need to have the same eigenvalues - "the distance with respect to O" is a notion that doesn't change if you translate your coordinate system around or rotate it.
A: To compare two coordinate systems, we first have to agree on what they describe. One can look at quantum states as points in an abstract space. Then, a quantum operator is a function completely defined by how it maps points (states) in a space, to other points (states) in a space (possibly the same). Furthermore, if we both do the same physical measurement process on the same state (point), we will get the same measurement result. We can now start to disagree on the description and physical interpretation of the states, operators and measurements, but we can always go back and agree on which points we are talking about, which functions we are talking about, and what measurement value we got.
As to your spesific example, say you use a coordinate system which is translated a distance $a$ with respect to my coordinate system. The point which in my coordinate system is described as $|x\rangle$, is in your coordinate system described as $| x+a \rangle$. Lets look at an abstract operator/function $\mathscr O$, which I will call $X$ in my coordinate system, and you call $X´$ in your coordinate system (they are related by $X´=UXU^{-1}$, where $U$ is unitary). Lets say that this particular operator obeys the equation $X|x\rangle=x|x\rangle$ in $my$ coordinate system (I would rightfully say that $X$ is what one generically calls the position operator, because it obeys the defining equation). We both agree on which operator, i.e. which function we are talking about: its the function $\mathscr O$ which takes the point which I describe as $|x\rangle$, and you describe as $|x+a\rangle$, to the point I describe as $x|x\rangle$ and you describe as $x|x+a\rangle$. We agree on which function it is, but we disagree on its physical meaning. In my coordinate system, its the position operator. In your coordinate system, its not, because instead it obeys the equation $X´|x+a\rangle=x|x+a\rangle$ (or equivalently $X´|x\rangle=(x-a)|x\rangle$). Lets now look at a complelety different operator, lets call it $Y´$ in your coordinate system (you consistently use primes to name things) and which in $your$ coordinate system satisfies $Y´|x\rangle = x|x\rangle$ (and equivalently, $Y´|x+a\rangle=(x+a)|x+a\rangle$). This operator is in fact what you would rightfully interpret to be the position operator - in $your$ coordinate system. In fact, the two operators are related by the equation $X´=Y´-aI$ ($I$ is the identity operator), and they are ceratinly not the same operator.
Regarding the use of primed notation:
Lets again denote the function we agree we are talking about, by $\mathscr O$. Keep in mind that a passive coordinate transformation only changes the descriptions, nothing else. Its important to use the primed notation, because the form and interpretation of $\mathscr O$ in our respective coordinate systems can be very different - the primed notation reminds us of that. We know that the two descriptions are related by $X´=UXU^{-1}$, so you can appreciate why they may look different. In the present example, in my coordinate system, $\mathscr O$ has a very simple form, its just $X$, "the position operator in my coordinate system". In your coordinate system, it has the form $Y´-aI$, "the operator which is the position operator in your coordinate system minus $a$ times the identity operator".
A: The quantum answer, after invoking Wigner, goes like this: consider a unitary transformation $U$. Applying it to any pure state yields $|\psi^\prime\rangle=U|\psi\rangle$. Applying it to any operator yields $A^\prime=U A U^\dagger$. Then
$$A^\prime |\psi^\prime\rangle=U A U^\dagger U |\psi\rangle=U \left(A|\psi\rangle\right).$$ This means that, if you want to know the action of $A$ on $|\psi\rangle$ in the new coordinates, you just transform the result you got in the old coordinates according to $U$. And, if you take expectation values, you find
$$\langle\psi^\prime |A^\prime|\psi^\prime\rangle=\langle \psi|A|\psi\rangle.$$ Keeping these relationships in mind helps me understand all of active and passive transformations.
Let's use a mundane example that you are trying to make a long piece of paper by taping together two pieces of paper, then you change which units you want to use for the measurement. The piece of paper is like $|\psi\rangle$, the action you do on the paper by taping two pieces together is like $A$, and changing units is like $U$. Initially your paper is 28 cm long, you apply the taping action ("add 28 cm"), now your paper is 56 cm long. In the transformed coordinates, initially your paper is 11 inches, you apply the taping action ("add 11 inches"), now your paper is 22 inches long. And the relationship that holds it all together is that you can apply the transformation to the final result and it all makes sense: transform 56 cm to inches and you find 22 inches.
To specifically answer your questions:
(a) If eigenvalues changed, it would make a difference if we applied the transformation first to $A$ and $|\psi\rangle$ and then did our action $A^\prime |\psi^\prime\rangle$, versus if we applied the transformation after the action as $U(A|\psi\rangle)$. With the piece-of-paper example, it shouldn't matter if we change our units of measurement before or after we tape two together two pieces of paper! We tend to take this freedom for granted in most physical theories (I can talk about how far away you are from me without specifying where the earth is in the solar system), and so Wigner's theorem codifies the only ways in which we can build this freedom into our physical theories.
(b) Yes, exactly. The density operator is an operator, and operators transform as $\rho\to U\rho U^\dagger$. In particular, it is a Hermitian operator with positive eigenvalues and we have just seen that the eigenvalues of an operator should be unchanged by our set of transformations. This works nicely because it just generalizes the notion of the transformations of states:
$$U\rho U^\dagger=U\left(\sum_m \omega_m |\psi_m\rangle\langle\psi_m|\right)U^\dagger=\sum_m \omega_m U|\psi_m\rangle\langle\psi_m|U^\dagger=\sum_m \omega_m |\psi_m^\prime\rangle\langle\psi_m^\prime|.$$ You'll find all of the nice relationships you want with $A^\prime \rho^\prime=U (A\rho)U^\dagger$, $\mathrm{Tr}(A^\prime \rho^\prime)=\mathrm{Tr}(A \rho)$, etc.

Now perhaps the confusion arises solely from notation. Ballentine uses $x$ and $|x\rangle$ to denote the eigenstate and eigenvalue of operator $X$. The prime, or the $a$ label in $x^\prime=x+a$, tells us about a new coordinate system. But maybe it would be better to label the ket and operator by $a$ but not the eigenvalue: $X_a$ has eigenstates $|x\rangle_a$ with eigenvalues $x$. Then you don't have to worry about the eigenvalue changing. Now, $X_a$ and $|x\rangle_a$ are the operator and eigenstate in coordinate system labelled by $a$, always satisfying $X_a|x\rangle_a=x |x\rangle_a$ for all $a$.
Or we just explicitly say that the eigenvalue is not labelled by the coordinate system, only the operators and states are labelled by the coordinate system, because the transformations only exist on operators and states (think $U xU^\dagger=x$ for scalar $x$). This must be the case, because most transformations involve more than one degree of freedom! Sure, it is easy to do $x\to x+a$, but how should some eigenvalue change if your transformation is "move left two and up three" or "rotate clockwise by $\pi/3$?" None of these transformations tells you what to do to some scalar eigenvalue, so the eigenvalues must be left unchanged by the transformations.
As an example, let's think classically about what operators, their eigenvectors and their eigenvalues are by considering rotations of a vector in 3D. Suppose you have an operator that rotates a vector by angle $\theta$ about axis $\mathbf{n}$. What are its real eigenvectors, restricting to real because we want vectors that actually point somewhere in 3D? The vectors that are unchanged by rotations: $\mathbf{n}$ and any other longer or shorter vector in that direction. What are its eigenvalues? The only real eigenvalue is 1, for any vector $\propto \mathbf{n}$. If you rotate the coordinate system, we change our labels for the direction $\mathbf{n}$. Since this label is important for both the operator and the eigenstate, they both must change, but since the label is not important for the eigenvalue, it can stay the same. Physically, the new coordinate system has a rotation around some other axis $\mathbf{n}^\prime$, the eigenvectors are vectors pointed along that axis $\mathbf{n}^\prime$, but the eigenvalue is still the same 1 because it just signifies a vector that is unchanged by the rotation. If we rotate the actual vectors instead of rotating the coordinate system, the same conclusion holds. The operator and eigenvector change because they rely on directions, but the eigenvalue does not change because it just tells us "that vector remains unchanged by that operation." This then all generalizes to any other physical system and to eigenvalues that are different from 1, but with no different intuition.
A: From a passive transformation point of view, this primed notation is not making much sense in my opinion. Think about classical mechanics on a phase space. The possible states of a system are abstract points $p$ on a symplectic manifold. Observables are functions $f(p)$. Now, suppose you choose some co-ordinate chart to describe this manifold. You will describe your states by some co-ordinates $(x, y) $ and your observables by some functions $g(x, y)$. If you change the co-ordinate system, your state becomes $(x', y') $ and your observable becomes $g'(x', y')$. The abstract state remains $p$ and the abstract observable remains $f(p)$, i.e. with no primes.
The ket notation $|\psi \rangle$ refers to an abstract vector. In the position basis, it may be written as $\psi (x)$. In a translated co-ordinate system, it may be written as a different function $\psi' (x) =\psi (x+a) $, in such  a way that the inner product $\int  \psi (x) \phi ^*(x)dx = \int dx \psi '(x) \phi '^*(x)dx $ remains invariant. But in the abstract, you will always write $|\psi \rangle$ and $|\phi\rangle$ for the vectors and $\langle \psi |\phi \rangle$ for their inner product. No primes get introduced.
My best guess is that the book was talking about active transformations.
Let's say you and your friend agree to do two identical experiments, but your friend will do it in a frame rotated relative to you.
Now, you will describe your experiment's initial state as some wavefunction $\psi (x)$. And you describe your friend's experiment's initial state as the rotated wavefunction $\psi '(x)$. In the abstract, we have:
$$|\psi '\rangle= R_{\theta} |\psi \rangle$$
Similar logic goes for observables. You describe your own $x$ co-ordinate observable as some operator $X$. But you describe your friend's $x$ co-ordinate observable as a rotated operator $X'$. In the abstract,
$$X'=R_{\theta}(X)=UXU^{\dagger}$$
Now, since you both are doing identical experiments, the expected value of your $x$ co-ordinate that you will measure in your experiment will be identical to the expected value of your friend's $x$ co-ordinate that they measure in their experiment:
$$\langle \psi |X|\psi \rangle=\langle \psi ' |X'|\psi' \rangle$$
It should also make sense why the eigenvalues are identical (which was your earlier question) :
$$X|\psi\rangle=x|\psi\rangle$$
imples:
$$UXU^{\dagger} U |\psi\rangle=U(x|\psi \rangle)$$
Or
$$X' |\psi '\rangle= x |\psi '\rangle$$
A: The best way to do this is the simplest one and the one which clarifies what the book says. I don't understand what the author says by similar operator as most of the others have assumed a translation operation (the easiest one) but it is easier to use a generalised version. so like @quantum mechanics says the expectation value should be some for both the operators but what he missed out is the relation between the eigenvalue and the expectation value. say ,
$A|\Psi_N\rangle=C_N|\Psi_N\rangle$ multiplying with the bra of $|\Psi_N\rangle$ you get
$\langle \Psi_N|A|\Psi_N\rangle=\langle\Psi_N|c_N|\Psi_N\rangle$
$\langle A\rangle=c_N$. (this is because for pure states you can assume normalised states)
Now as the books says after a Unitary transformation you should get $$\Psi'_N=U\Psi_N $$and $$ A=UA'U^\dagger$$ Usually we take the opposite and replace the A with A' but since Unitary operators and their conjugate's product is Identity, you can show either fomulation is equivalent.
For the new state for say, some new eigenvalue $c_N'$ $$A'|\Psi'_N \rangle=c_n'|\Psi_N' \rangle$$
Following a similar process as the first one by multiplying the bra of $|\Psi_N'\rangle$ you get ;
$$\langle\Psi_N'|A'|\Psi_N'\rangle=\langle\Psi_N'|c_N'|\Psi_N'\rangle$$
Applying the Unitary transformation
$$\langle\Psi_N|U^{\dagger}A'U|\Psi_N\rangle=c'_N\langle\Psi_N|U^{\dagger}U|\Psi_N\rangle$$
But as $\Psi_N$ is normalised you should get;
$$\langle \Psi_N|A|\Psi_N\rangle=c'_N$$
But we already know that $\langle A \rangle=c_N$
Hence,
$$c_N=c'_N$$ and similar operators have same eigenvalues
A: The symbol $\lvert{P\rangle}$ means "the particle is at the position $P$ in space", i.e.,
$$
\lvert{P\rangle}~~~~~~\mbox{is shorthand for}~~~~~~\lvert{\mbox{particle is at position }P\rangle}\,.
$$
Choose a coordinate system, with $x=0$ as the origin, and suppose that the value of the coordinate $x$ at position $P$ in this coordinate system is $x_P$. Usually, we would label this state as $\lvert x\rangle$, but I am trying to make it clear that the state of the particle should be independent of our labeling/choice of coordinate system. So, instead of labeling the state with the value of its position in the coordinate system, I am labeling the value of the coordinate with the position.
Perform an active translation (by $a$) of the system---which is to say, actively move the particle to some other position in space---in which case the new state of the system is
$$\hat{T}_a\lvert{P\rangle} = \lvert Q\rangle\,,$$
which means that "the particle is now at the position $Q$."  Since we are translating by $a$, the value of the position at $Q$ is
$$
x_Q = x_P+a\,.
$$
Now, suppose we measured the position of the particle before and after the transformation. Since it is a measurement of position, it is necessarily tied to a choice of coordinate system, and so it might as well be the one that we have already chosen. The first result must therefore be $x_P$.  Then, after translating the system by $a$, we must get $x_Q=x_P+a$ as the result.$^1$
$^1$ We are ignoring the Schrodinger evolution that inevitably results in the particle evolving into a state of ill-defined position. I suppose we could assume that we have confined the particle to a very narrow trap (maybe it's a molecule trapped by optical tweezers!), and we are physically moving the entire apparatus.

At this point, we create a linear operator $\hat{x}$ representing the position of the particle (i.e., a self-adjoint operator corresponding to a dynamical variable). But we have to be careful (as mentioned above), because its possible values are dependent on the coordinate system we chose at the beginning. That is to say, if we act with $\hat{x}$ on the pre- and post-translation states, we'll get the pre- and post-translation positions expressed in our current coordinate system, i.e.,
$$
\hat{x}\lvert P \rangle = x_P\lvert P  \rangle\,,~~~~~~~~
\hat{x}\left(\hat{T}_a\lvert P \rangle\right) = \hat{x}\lvert Q \rangle
=x_Q\lvert Q \rangle=(x_P+a)\lvert Q \rangle\,.
$$
(Really, we should be talking about the expectation values of $\hat{x}$ to do this correctly, because then we can talk correctly about measurements, but I think this is fine here.)

All of that is fine. Since we're not changing our coordinate system, none of our operators change, but the state of the particle does and hence so does the value returned by a measurement of the position. But now let's do the opposite. Let's instead translate the origin of our coordinate system by $-a$ while keeping the state of our system fixed.  Under this passive transformation,
$$
\lvert P \rangle~~~~~~\mbox{stays as}~~~~~~\lvert P \rangle\,.
$$
However,
$$
\hat{x}\lvert P \rangle = x_P\lvert P \rangle~~~~~~\mbox{must somehow become}~~~~~~\hat{x}\lvert P \rangle = (x_P-a)\lvert P \rangle\,,
$$
but this doesn't make sense, since these can't both be true at the same time. So, instead, we define a new position operator $\hat{x}'$ that measures position with respect to the new coordinate system, i.e.,$^2$
$$
\hat{x}\lvert P \rangle = x_P\lvert P \rangle~~~~~~\mbox{becomes}~~~~~~\hat{x}'\lvert P \rangle = (x_P+a)\lvert P \rangle\,.
$$
Since the position kets $\lvert P \rangle$ are a complete basis, this means that the old and transformed operators are related by
$$
\hat{x}' = \hat{x}+a\,,
$$
and we have established our transformation rule for the position operator under a translation.
To complete this picture, let's translate the system by $a$, act with $\hat{x}$, then translate back, i.e.,
$$
\hat{T}_{-a}\hat{x}\hat{T}_a\lvert{P\rangle}
=
\hat{T}_{-a}\hat{x}\lvert{Q\rangle}
=
\hat{T}_{-a}x_Q\lvert{Q\rangle}
=
x_Q\hat{T}_{-a}\lvert{Q\rangle}
=
x_Q\lvert{P\rangle}
=(x_P+a)\lvert{P\rangle}\,.
$$
The action of $\hat{T}_{-a}\hat{x}\hat{T}_a$ on $\lvert P\rangle$ is the same as the action of $\hat{x}'$. We have established the operator transformation rule
$$
\hat{x}'=\hat{T}_{-a}\hat{x}\hat{T}_a = \hat{x}+a\,.
$$

Note that in the above, we either transform the state (an active transformation that corresponds to the physical act of "moving" the system) or we transform the operator (a passive transformation that corresponds to choosing a different coordinate system). So what is all this about transforming the operators and the states at the same time?  There are two possibilities, and while the OP has me leaning toward the former, I think the latter one has some merit, too (at least for the purposes of understanding):

*

*Ballentine is performing both an active transformation of the system by "picking it up and moving it" and a passive transformation to carry the coordinate system along with the system.


*Ballentine is playing fast and loose by identifying states and the representations of the states in a basis.
Simultaneous active and passive transformations
The OP suggests that Ballentine is performing both an active transformation of the system by "picking it up and moving it" and a passive transformation to carry the coordinate system along with the system. (This last part is my  interpretation.) Let's examine the translation situation as above. To translate the system, we transform the physical state of the system as
$$
\hat{T}_a\lvert P \rangle = \lvert Q \rangle= \lvert P' \rangle\,.
$$
(I'm mimicking Ballentine's notation here by priming the variable $P$, but I like to use a different symbol $Q$ to represent the fact that this really is a different state corresponding to a different position). Let's also transform the coordinate system in such a way that it moves with the system. In this case that means that we are making the transformation
$$\hat{x} \to \hat{x}'=\hat{x} - a
$$(opposite to what we did above when we were trying to show the equivalence between an active transformation in one direction and a passive transformation in the other direction). The action of the passively transformed operator on the actively transformed vector is
$$
\hat{x}'\lvert P' \rangle = (\hat{x} - a)\lvert P' \rangle
= ({x}_{P'} - a)\lvert P' \rangle = (x_P + a -a)\lvert P' \rangle  = x_{\textrm{P}}\lvert P' \rangle \,.
$$
The effect is to leave the eigenvalue the same.
A similar argument illuminates his second point. Start with a vector $\lvert \psi \rangle$ expanded in the position basis as
$$
\lvert \psi \rangle = \sum_P \lvert P \rangle \langle P| \psi\rangle\,,
$$
ignoring the vagaries of a continuous basis for the sake of argument. The probability of getting the value $x_{\textrm{P}}$ upon measuring the position as measured in the old coordinate system is $|\langle P | \psi \rangle|^2$, since $\hat{x}$ is the "old" coordinate position operator, and $\hat{x}\lvert P \rangle = x_P\lvert P \rangle$.
Now, we actively transform this state as
$$
\lvert \psi' \rangle = \hat{T}_a\lvert \psi \rangle = \hat{T}_a\sum_P \lvert P \rangle \langle P| \psi\rangle
= \sum_P \hat{T}_a\lvert P \rangle \langle P| \psi\rangle
= \sum_P \lvert P' \rangle \langle P| \psi\rangle
\,.
$$
The probability of getting the value $x_{\textrm{P}}$ upon measuring the position as measured in the new coordinate system is $|\langle P' | \psi' \rangle|^2$, since $\hat{x}'=\hat{x}-a$ is the "new" coordinate position operator, and $\hat{x}'\lvert P' \rangle = x_P\lvert P' \rangle$.  If the physical system is translationally invariant, then these probabilities must be the same (because we've translated the coordinate system along with the system!).
But---and here is the part that is unsatisfying to me---these are automatically the same because$^4$
$$
\langle Q'\lvert \psi' \rangle = \langle Q' |\sum_P \lvert P' \rangle \langle P| \psi\rangle
=  \sum_P\langle Q' \lvert P' \rangle \langle P| \psi\rangle
=\sum_P\delta_{QP}\langle P| \psi\rangle = \langle Q | \psi \rangle
\,.
$$
Not only are the probabilities the same, the amplitudes are the same.  Perhaps I have assumed unitarity somewhere, but if not, it sure seems like we haven't done anything. In other words, this equality of probabilities should be a physical constraint on the formalism that forces unitarity of the transformation operators, but at the same time it seems to be satisfied automatically.
Perhaps the explicit form of the translation operator assumes unitarity, and so I am merely illustrating the idea rather than proving it for this special case, but it's not clear to me how the argument (or "illustration") would really be all that different if I proceeded in more generality.
All that said, this sure seems to be the approach Ballentine is implicitly assuming.  That said, the unsatisfying nature of the (lack of) constraint provided by the equality of probabilities makes it so the other interpretation, outlined below, is also a possible way to think about what's going on here.
$^4$ Since $Q$ and $P$ both represent positions in space, the inner product of the corresponding states is only zero if $Q$ and $P$ are the same position. Similarly, since we are translating by the same amount $a$, $Q'$ and $P'$ are the same position if and only if $P$ and $Q$ are, and so $\langle P' | Q' \rangle \neq0$ if and only if $\langle P | Q \rangle \neq0$.
States and operators versus representations of states and operators
The other interpretation is that Ballentine is playing fast and loose by identifying states and the representations of the states in a basis.$^3$ The following somewhat recapitulates what I wrote in this answer.
When we change the basis we are working in, we change the matrix representations of a vector while keeping the interpretation of the vector. That is, if we write a state $\lvert\psi \rangle$ in two different bases as
$$
\lvert \psi \rangle = \sum_n \lvert v_n \rangle \langle v_n| \psi\rangle
=
\sum_j \lvert \varphi_j \rangle \langle \varphi_j |\psi\rangle\,,
$$
then we can identify the two column vector representations of the state $\lvert\psi\rangle$ as
$$
\lvert\psi\rangle_{v\textrm{ basis}}\to
\psi_{\textrm{in basis }v}
=
\left[\begin{array}{c}
\langle v_1| \psi\rangle\\
\langle v_2| \psi\rangle\\
\langle v_3| \psi\rangle\\
\vdots
\end{array}
\right]\,,~~~~~~~~~
\lvert\psi\rangle_{\varphi\textrm{ basis}}\to
\psi_{\textrm{in basis }\varphi}
=
\left[\begin{array}{c}
\langle \varphi_1| \psi\rangle\\
\langle \varphi_2| \psi\rangle\\
\langle \varphi_3| \psi\rangle\\
\vdots
\end{array}
\right]\,.
$$
These represent the same state but are obviously not equal to each other. If we are transforming from the $v$ basis to the $\varphi$ basis, might call the first $\psi$ and the second $\psi'$, indicating that we have "transformed" the state, but we really haven't ($\lvert \psi \rangle$ is $\lvert \psi \rangle$). At the same time, the matrix representations of the operators change, too, because $\langle v_i | \hat{A} | v_j\rangle \neq \langle \varphi_i | \hat{A} | \varphi_j\rangle$. Thus, both "states" and "operators" change in this picture, but what we really should say is that the representations of these objects are the things that are changing.
Then, of course, if $\lvert \psi\rangle$ is an eigenvector of $\hat{A}$ with eigenvalue $a$, then
$$
\hat{A}\lvert \psi\rangle = a\lvert \psi\rangle\,,
$$
no matter what, and so it also must be true that
$$
A_{\textrm{in basis }v} \psi_{\textrm{in basis }v} = a \psi_{\textrm{in basis }v}\,,
~~~~~~~~~
A_{\textrm{in basis }v} \psi_{\textrm{in basis }\varphi} = a \psi_{\textrm{in basis }\varphi}\,,
$$
or, more pointedly,
$$
A\psi = a\psi~~~~~~\mbox{and}~~~~~A'\psi' = a\psi'\,.
$$
Ballentine goes on to talk about what is essentially the inner-product preserving property of unitary transformations, but a basis change between orthonormal bases is unitary, and in that case the inner product between two states and their transformed versions must be the same.
This last part is probably unsatisfying. All I can say is that once you're working with representations, you have to change the "states" (read: representations) under a transformation of coordinates because this is synonymous with a change of basis. But then, since you need to have equality of probabilities, because you haven't changed the state, only its representation, the result follows that the squares of those coefficients need to be the same. So that's possibly further evidence for the take that Ballentine is really talking about representations of vectors/operators and not the objects themselves.
$^2$ It's $x_P+a$ because we've shifted the origin to the left by an amount $a$. Suppose that point $P$ is to the right of the origin; then it should end up farther from the origin in the new coordinate system, and hence its value in the new coordinate system should be larger. 
$^3$ He is not alone; Shankar's discussion of this translation stuff is done correctly I think, but he isn't careful about states versus representations of states earlier in his book when first discussing transformations. These are two excellent textbooks written by two physicists who have spent a lot of time thinking very deeply about quantum mechanics. Perhaps it is me that is wrong here, but to the extent that coordinate transformations can be brought about via basis changes, then what follows seems reasonable.  And note that there is a correspondence between unitary operators and change-of-basis transformations. One shouldn't think about them in quite the same way, but they have equivalent matrix representations.

Some of this confusion could possibly be mitigated by a better choice of notation, which is what I tried do here by labeling the position state with the name of a point in space rather than the value of its coordinate in a coordinate system. I think we're stuck with  $\hat{x}\lvert x\rangle = x \lvert x\rangle $, though, and since there is great utility in labeling/indexing eigenvectors by their eigenvalues, I don't think it's such a bad thing. We just need to be more careful in how we talk about things, especially mixing up identifying objects and their representations.  This becomes pretty maddening in quantum field theory, where there are multiple layers of objects: coordinates (which no longer correspond to operators), fields defined on those coordinates (which are operators), Lorentz transformations (whether active or passive) and how they act on coordinates and then Lorentz transformations and how they transform fields and then how they transforms states!, etc. In addition, it is often unclear (at least for me) whether we are performing active or passive transformations or both!
