Looking at the table of specific heat capacities on Wikipedia, there is quite a wide range of heat capacities for the solids listed and many are much less that $1~\mathrm{kJ}~\mathrm{kg}^{-1}~\mathrm{K}^{-1}$. The Dulong-Petit law provides an approximate limit of $25~\mathrm{J}~\mathrm{mol}^{-1}~\mathrm{K}^{-1}$ for the molar heat capacity of atomic crystals (and as you can see, predicts the molar heat capacities of many metals pretty well), but makes no statements about mass heat capacity.
As far as solids with very high specific heat capacities go, try long chain hydrocarbons (paraffin and polyethylene are listed) or water ice, as well as anything that has a lot of water in it. For volumetric heat capacities, metals become competitive because of their typical high densities.