So to my understanding as a layman is that energy transfers from one material to another (I guess that's why there's potential and kinetic energy), for example photons to solar panels. Now my question is let's say you have no material in the universe where will all the energy go? I mean will it exist on its own? And what form will it be in? Or is energy just there independent of form?
So to my understanding as a layman is that energy transfers from one material to another
Not really. Energy is an attribute of a system in a particular configuration. That system might be, for example, the Sun and its planets; the Earth and a falling stone; the molecules that make up a volume of gas; the protons, neutrons and electrons that make up an atom; the quarks inside a proton etc. We give labels to the different components of energy within a system - kinetic energy, potential energy, chemical energy, nuclear energy etc. - but it is all just energy.
Let's say you have no material in the universe where will all the energy go ?
If "no material" means no matter, no radiation, absolutely nothing except for space and time themselves then there is nothing to have the attribute of energy, so there is no energy. However, the idea of a completely empty universe is somewhat problematic in quantum mechanics - to the best of our knowledge the most "empty" state that can exist is a quantum vacuum, but even in that state quantum fields still exist and have a non-zero amount of energy called vacuum energy. This may (or may not) be related to the dark energy that appears to be driving the expansion of our universe.