Can eigenstates of a Hilbert space be thought of as delta functions? Say we have an observable that describes a Hilbert space and that observable acts on state kets. Lets take the position observable for example. Then $\langle y|x\rangle = \delta(y - x)$. But can the eigenstates of the position observable be individually thought of as delta functions? 
$$ A |x\rangle = x'|x\rangle $$
Is this $|x\rangle$ then individiually a delta function picking $x'$ out of $A$? Wouldn't this also imply that we have an infinite number of delta function eigenstates in the observable space?
 A: 
But can the eigenstates of the position observable be individually thought of as delta functions?

Yes they can in a sense but it is rather inaccurate. First, kets and functions(distributions) are somewhat different things although they share most of their properties. If the ket $|x'\rangle $ satisfies
$$
\hat{x} |x'\rangle = x' |x'\rangle,
$$
then we say this ket is generalized eigenvector of the operator $\hat x$.
This ket can be represented in the space of distributions on coordinate $x$ by delta distribution located in $x'$, since in distributive sense
$$
x \delta(x-x') = x'\delta(x-x'),
$$
which has the same structure as the equation above. The difference between these two equations is that the first is stated without using any special coordinate; neither $x$, nor $p$ is used. The second used coordinate $x$. So kets and distributions are not the same things; the kets can be thought of as "coordinate-free" notation for the distributions.
Short note to the use of word "eigenstate" in this context:
delta distribution is not normalizable and thus does not belong to the set of distributions describing physical state in the sense of the Born interpretation. So it is not very good to call it or the corresponding ket $|x'\rangle$ an eigenstate of the position operator. It is better to call it "improper eigenfunction" or "improper eigenvector".

Wouldn't this also imply that we have an infinite number of delta function eigenstates in the observable space?

There is infinite number of distinct delta distributions defined on $x$. But they are not "states"; they do not belong to the Hilbert space $L^2(R)$.
A: In general, assume that $\mathcal{H}$ is a separable Hilbert space and that $A:\mathcal{H}\rightarrow\mathcal{H}$ is a linear operator. The hilbert space $\mathcal{H}$ being separable, admits a countable basis, which we can write as $\eta_i$ with $i\in\mathbb{Z}$ (these are not necessarily the $\delta$ functions that you speak of, but $\mathcal{H}$ being separable is isomorphic to $\ell^2(\mathbb{Z})$, in which case each $\eta_i$ can be mapped onto $\delta_j\in\ell^2(\mathbb{Z})$, where $\delta_j$, $j\in \mathbb{Z}$, are honest $\delta$ functions. 
Now assume that $A$ admits eigenvectors $e_j\in \mathcal{H}$, $j\in\mathbb{Z}$, such that the following is satisfied. 
1) $\{e_j\}_{j\in\mathbb{Z}}$ are linearly independent (in the sense that any finite subset forms a set of linearly independent vectors.
2) $\{e_j\}_{j\in\mathbb{Z}}$ is complete in the sense that the set of all finite linear combinations of elements from $\{e_j\}$ forms a dense subset of $\mathcal{H}$.
Then we see that $\{e_j\}$ forms a complete bases of $\mathcal{H}$. After normalizing each $e_j$ (or we can assume that they have already been normalized) we can now unitarily map everything into $\ell^2(\mathbb{Z})$, by mapping $e_j$ onto $\delta_j$. This is possible if and only if conditions (1) and (2) are satisfied.
In other words: every linear operator on a separable Hilbert space can be unitarily mapped to an operator on $\ell^2(\mathbb{Z})$. In this case, if the operator admits a set of eigenvectors that forms a complete basis, then this set can be mapped onto honest delta-functions of $\ell^2(\mathbb{Z})$. 
Of course, if condition (2) is relaxed, then one can instead consider a subspace of the original Hilbert space which is the closure of the set of all finite linear combinations of the eigenvectors. The operator would also have to be bounded (equivalently, continuous) to pass from a dense subset to its closure. 
A: Okay, so a general observable acting on $|x\rangle$ won't give you $x' |x\rangle$. Only the position operator, acting on the state $|x'\rangle$ will give us $x'|x'\rangle$, where the x' is a label for the state, think of it as a number, not a variable. Just because the state $|x'\rangle$ is an eigenstate of the position operator, it does not mean that it necessarily an eigenstate of any other general operator. 
$|x'\rangle$ is a state, we want it to be a position eigenstate. What that means is that if I have a particle in the state $|x'\rangle$, then the wavefunction written as a function of position better look like a delta function around the point $x'$. But remember that I could also write the wavefunction as a function of momentum or some other operator. The statement
$$ \langle y|x \rangle = \delta(x-y) $$
says just that. Think about $\langle y|$ as a function that takes in a vector and gives you a number, that number is the value of the wavefunction of the state represented by $|x'\rangle$ written as a function of position at the point $x'$.
If we want to think of space as continuous, then we'll need every point to be an eigenstate of the position operator. So just like there's infinite points in space in classical physics, there are infinite delta function eigenstates of the position operator in quantum mechanics. But in quantum mechanics you think of the function $y(x)$ instead in terms of vectors so that notationally
$$y(x) = \langle y | x \rangle .$$ 
Right, that's why we can use dirac notation to talk about functions.
