All matter we observe around us is composed by atoms and molecules, and atoms and molecules are composed of elementary particles , seen in this table, where the photon is a part of.
The classical electromagnetic light, at the particle level, emerges from a superposition of zillions of photons. Photons interact with the electromagnetic interaction with the electrons around the nuclei of the atoms and with the collective spill over electric and magnetic fields of solids, liquids and gases. Matter may be macroscopically neutral, but the photons which compose a light beam can interact at the particle level of the components of matter. Thus a wall ( or a mirror) is neutral, but light reflects from the wall(mirror), the photons of light scattering from the wall(mirror) at the level of the constituent particles and the topology of their electric and magnetic fields.
Does this imply that neutrons won't have any interaction with light
Neutrons as you observe, are composed of quarks, which as quantum mechanical point elementary particles have a wavefunction. There will always be a probability for a photon to interact with a quark through the electromagnetic coupling, particularly a high energy one. There will be similar diagrams with quark loops as shown in my answer here.