What is the physical meaning of $a_{\vec{p}} \! \mid \! 0 \rangle$ $a^\dagger_{\vec{p}} \! \mid \! 0 \rangle = \mid \! p \rangle$ is interpreted as a creation of a particle with momentum $p$ from the vacuum. $a_{\vec{p}} \! \mid \! p \rangle = \mid \! 0 \rangle$ is interpreted as the annihilation of a particle with momentum $p$. But in the textbooks whenever the annihilation operator is applied to the vacuum itself something I don't really understand happens:
$$a_{\vec{p}} \! \mid \! 0 \rangle=0$$
I think I know how to distinguish between $0$ and the vacuum state: I assume when we just write $0$ we mean the zero vector in the Fock space, whereas the vacuum state is the lowest energy nonzero vector. However, whenever the creation or annihilation operators are applied to higher energy states there seems to be a very straightforward physical interpretation: a particle is created or destroyed. I don't really understand what the physical interpretation of annihilating the vacuum is. 
 A: It means it's "the end of the line". The vacuum state is, as you correctly say, not the zero state. It has energy content, and physical meaning - it's the state with no particles. Annihilating the vacuum leaves...nothing. Trying to take a particle out of it is not possible - it gives you the zero vector, which does not represent a physical state, since it is not a member of the projective Hilbert space. In fact, $a\lvert 0 \rangle = 0$ is the definition of the vacuum - the vacuum is the state you cannot take any particle out of.
A: $a^\dagger a\! \mid \! n \rangle = n\mid \! n \rangle$. For vacuum you can write $a^\dagger a\! \mid \! 0 \rangle = 0\mid \! 0 \rangle$, if you like. No need to write 0 solely.
A: There are already good answers. Here we merely use different words and stress different things. 


*

*The $n$'th Fock space ${\cal H}_n$ is the vector space spanned by $n$ particle states. 
In other words, if we omit the zero-vector, then ${\cal H}_n\backslash\{0\}$ is the space of $n$ particle states. Be aware that we will often casually refer to ${\cal H}_n$ as the space of $n$ particle states, although the zero-vector $0$ is not an $n$ particle state. 

*For negative $n<0$, let us define ${\cal H}_{n<0}:=\{0\}$ to be the zero vector space.

*The Fock space $${\cal H}_0~=~ \mathbb{C} |\Omega \rangle$$ of $0$ particles is spanned by the vacuum state$^1$ $|\Omega\rangle$. Be aware that the vacuum state $|\Omega\rangle$ is often written as $|~ \rangle$ or $|0\rangle$. The latter should not be confused with the 1 particle state $|\vec{p}\!=\!\vec{0}\rangle$ nor the zero-vector $0$.

*The annihilation operator 
$$ a_{\vec{p}}:{\cal H}_n\to {\cal H}_{n-1}$$ goes from the Fock space ${\cal H}_n$ of $n$ particles to the Fock space ${\cal H}_{n-1}$ of $n\!-\!1$ particles. It removes a particle with momentum $\vec{p}$. If none of the $n$ particles have
momentum $\vec{p}$, then the result will be the zero-vector $0$; not an $n\!-\!1$ particle state.

*OP's three equations can alternatively be written as 
$$a^{\dagger}_{\vec{p}}  | \Omega \rangle ~=~ |\vec{p}\rangle, \qquad a_{\vec{p}}  | \vec{p} \rangle ~=~ |\Omega\rangle, \qquad a_{\vec{p}}|\Omega \rangle~=~0, $$
respectively.
--
$^1$ Note that the notions of vacuum $|\Omega\rangle$ and particle number depend on the choice of creation and annihilation operators. They may not be invariant under e.g. Bogoliubov transformations, and non-perturbative effects. See also the notion of ground state.
