Dirac Notation and Coordinate transformation of a function Lets say we have a 1 dimensional system with coordinate $x$ and the associated operator $\hat x$ with eigenstates $|x\rangle$. A function of $x$ is defined as
$$
f(x) = \langle x |f\rangle \tag{1}
$$
Now we introduce a new coordinate $y$ with operator $\hat y $ and corresponding eigenstates. x and y are describing the same 1 dimensional space. and both sets of eigenstates are assumed to be complete and orthogonal in the the sense of
$$
\hat 1 = \int dx ~|x\rangle \langle x|\\
\langle x|x'\rangle = \delta(x-x').\tag{2}
$$
The relationship between coordinates $x$ and $y$ is given by
$$\begin{aligned}x(y) &= ay  \\
y(x) &= a^{-1}x
\end{aligned}\tag{3}$$
where $a$ is a real constant.
Due to completeness the function $f(x)$ can be expressed as
$$
f(x) = \int ^\infty_{-\infty}dy \langle x|y\rangle \langle y|f\rangle\tag{4} 
$$
What is $\langle x|y\rangle$ in this case? What is the relationship of $|x\rangle$ and $|y\rangle$? How would all of this apply to an example function like $f(x)= x^2$?
Does the coordinate transformation define these relationships or do i need more definitions to answer these questions?
I tried the following
$$
f(x(y)) = \tilde f(y) = a^2y^2 = \langle y|f\rangle\\
x^2 =  \int ^\infty_{-\infty}dy \langle x|y\rangle a^2 y^2\\
\Rightarrow \langle x | y \rangle = a^{-2}\delta(x-y)\tag{5}
$$
but that can't be right. The result depends on the function that i plug in, which shouldn't be the case. If had used a different $f(x)$ i would have gotten a different result for the matrix element $\langle x |y\rangle$. Where do i go wrong ?
My goal is to properly understand how a variable substitution, done in position representation, is expressed in Dirac Notation. Preferably starting in position representation and going "backwards" to Dirac Notation. I am uncertain how such a transformation is properly expressed in the more abstract Dirac representation or how it affects the states. I have no trouble doing a coordinate substitution in a integral of the form
$$
\langle f|f\rangle = \int f^*(x)f(x) dx \tag{6}
$$
for the right side of the equation which is given in position representation but i fail at doing the same in a more abstract manner.
The right side simply transforms to
$$\begin{aligned}
\int \langle f|x\rangle \langle x|f\rangle dx &= \int f^*(x(y))f(x(y)) dx(y)\\
? &=\int  f^*(y) f(y) |\frac{dx}{dy}| dy\\
\end{aligned}\tag{7}$$
On the other hand i can write the integral also as the following, using the completeness of $y$,
$$\begin{aligned}
\langle f | f \rangle &= \int \langle f | y \rangle \langle y | f \rangle dy \\
\end{aligned}\tag{8}$$
Now i set both equations expressed in $y$ equal,
$$
\int f^*(y) f(y) |\frac{dx}{dy}| dy = \int \langle f | y \rangle \langle y | f \rangle dy\tag{9}
$$
which leads me to a contradiction, under the assumption that $f(y)=\langle y | f \rangle = f(x(y))$. Is the error the assumption
$$
f(y)=\langle y | f \rangle = f(x(y)) \tag{10}
$$
or something else ?
 A: *

*Under a coordinate transformation $y=f(x)$, where $f:\mathbb{R}\to\mathbb{R}$ is a diffeomorphism, the position ket & bra transform as half-densities
$$\begin{align}|f(x)\rangle~=~&|f^{\prime}(x)|^{-1/2}|x \rangle,\cr \langle f(x)|~=~&|f^{\prime}(x)|^{-1/2}\langle x|,\end{align}\tag{A}$$
so that the completeness & orthogonal relations (2) transform covariantly.


*Phrased equivalently, the wavefunction
$$ \psi(x)~\equiv~ \langle x|\psi \rangle \tag{B} $$
transforms as a half-density under coordinate transformations.


*We can then construct a half-form object
$$ \psi(x)\sqrt{\mathrm{d}x} \tag{C}$$
that is invariant under coordinate transformations. A similar construction is used in e.g. geometric quantization.
A: I'm not sure what hack you'd be asking for in your comment to the good answer. You are meant to apply the transformation laws of the impeccable answer already provided to your scaling, (3), a change of basis,
$$
|y\rangle =  \sqrt{a}  |x\rangle,~~ \leadsto ~~\langle x|y'\rangle =  \sqrt{a} \langle x |x'\rangle. 
$$
The sensible version of your (4) then reads
$$
f(x)= \int dy'~ \langle x|y'\rangle \langle y'|f\rangle=a^{-1/2}\int dy' ~\langle y|y'\rangle \langle y'|f\rangle = a^{-1/2} \langle y|f\rangle,
$$
so your (5) and (10) are wrong. Recall what you do with coordinate and momentum space wavefunctions. You may write their correct, consistent, versions by the above, or for your example. Do you want me to do this?

In response to comment. For your simple scaling (3), recall
$$
\hat{y}|y\rangle= y|y\rangle=x/a|x/a\rangle=\hat x /a|x/a\rangle, \leadsto \\
= \hat x |x/a\rangle=x|x/a\rangle, \leadsto \\
|x/a\rangle\propto |x\rangle.
$$
What is the proportionality constant?  $|y\rangle =  \sqrt{a}  |x\rangle$, so
$$
\langle y|y'\rangle=\delta (y-y')=a \delta (x-x')=a\langle x|x'\rangle, 
$$
and $dx=a~dy$, as in the good answer.
