If you collide two ideal billiard balls, then that would be what you would call a perfectly elastic collision.
If you have a large dense collection of billiard balls, and you slam a new one into the collection, then there are a whole lot of elastic collisions, transfering energy and momentum in many ways that you would be hard-pressed to calculate exactly.
It you think about it statistically, that is thermal energy.
If some of the balls in the dense collection are tied together, then you get spinning pairs (rotational kinetic energy).
If some of the balls have non-zero angular moment of inertia, then some of the balls will spin (rotational kinetic energy).
If some of the balls are in groups of three, connected by springs, then they could end up vibrating (kinetic plus potential energy).
This is the way atoms and molecules behave, giving rise to thermodynamics.
So if you collide two beanbags together, making an inelastic collision, I'm inclined to agree with you - that's really just a zillion elastic collisions we can't compute.
So we just call it heat.
Of course, that's all classical. If you want to get into quantum stuff, @Anna's answer has it covered.