The case for momentum of a system to be conserved is that no external force should be acting on the system. This comes from newtons second law.
On alaysing the bullet rod system there are 2 forces, that acts on it::
Gravity- if we conserve momentum at the time just before the bullet hits the rod, to the momentum of the system just after the bullet collides with the rod, we can safely neglect gravity. This is because the time of collission is very small, subsequently the change in momentum of the system due to gravity is also very small. ( impulse or change in momentum is force times time.)
Hinge force - this is the force acting on the system at the point of contact of the rod with the horizontal wall. In the diagram it is a sort of triangular structure. It exerts force on the system, specifically normal and frictional force. Both these forces are impulsive and their effect on the net momentum of the system cannot be ignored. This is because, these contact forces take very large values at the time of collision, (thus they are impulsive) offsetting the small time period of collision. Hence momentum of the system cannot be conserved.
Contact forces between the bullet and rod are internal forces and cannot change the moment of the system. ( comes from newtons third law)
Finally to solve the question, you could try
Conserving angular momentum at the hinge of the rod. No force produces torque and so angular momentum is conserved.