The resolving power of a prism is given by the formula
$$ \frac{\lambda}{\Delta \lambda} = b\ \frac{dn}{d\lambda},$$
where $b$ is the base length of the prism, $\lambda$ is the wavelength and $n(\lambda)$ is the refractive index.
You don't say, but let's assume you are using a crown glass prism. According to this useful document, crown class has $dn/d\lambda \simeq 400$ cm$^{-1}$ in the yellow part of the spectrum. So, for a prism of baselength $b=5$ cm, you have a resolving power of about 2000 and hence in the yellow part of the spectrum you should be able to resolve lines with a separation of 0.25 nm. This should easily be sufficient to see H$\alpha$, the sodium D lines etc.
You have not said how you are focusing the light? I suspect what you are doing wrong is you have not built yourself a spectrograph; something like the image below (from here). You need to focus the dispersed light and you also need to make sure you have a slit-like aperture for best results.
If you don't do something like this, then effectively what you have are heavily overlapping images of the face of the prism at each wavelength. The overlap is likely to be so large that it destroys your resolving power. A lens can focus these images so that light of a particular wavelength falls at one spot on your screen.
EDIT: As a thought experiment, consider putting a filter in front of the prism that only lets red light at a particular wavelength through. What will the projected image look like in your current setup? My guess is that it will look like a red rectangle, with a width that is proportional to the size of the prism face that the light exits from.
Now change the central wavelength of the filter by 1 nm. The position of the projected rectangle will change slightly, but not by anywhere near the width of the rectangle, so that its image would overlap almost totally with the previous image. i.e. Light separated by only 1 nm is smeared together and cannot be resolved.
What you need to do with the lens is focus the rectangle down so that ideally it is just a very narrow slit-like image. That way a change in wavelength of 1nm will produce a clearly separated image.