Before I start, I should say, I am yet to meet a physicist who has a clue about the practical aspects of loudspeaker design. Likewise about audio engineers ;-) so the fact you are asking this, means you are well set up for making some good speakers.
John Bird has written an electronics textbook which could be very useful for you which covers all the introductory electronic engineering theory to get an idea of what its about. Great way to get started, but maybe too simple for you. Have a look.
Undergrad Electronics is actually not so deep as it is broad. So, if all you want to understand speaker design you actually won't need a lot (to get started anyway) and actually making something that works, which is always a good place to start.
There are many very good textbooks for electronics and loudspeaker design knocking about and the core principles from decades ago still apply today, so that is lucky. There is also a electronics stackexchange site if you have specific questions. Don't be afraid to look at the older textbooks. Quite often they have a lot more core principles in them then the modern books and the added bonus that they are much cheaper on the whole.
There are a plethora of related coursera.org courses on the topic so that might help you along.
You will need to know some tricks like nodal analysis, applying Kirchoffs laws, Thevinins theorem, applying Ohms law (obviously), small signal analysis (for operational amplifiers) and if you are going digital, logic gates, boolean algebra all the books on digital electronics tend to be the same more or less. Perhaps you already know that much already without me telling you so I guess where it gets more strictly into the physics is when you consider things like the quantum hall effect and electromagnetism in general (try Griffin), you will need to understand fourier series and fourier transfer and waves in general -how they behave at boundaries, how sound travels through the air, resonance the wave equation. It wouldn't hurt to learn a bit of thermodynamics. Your carefully crafted loudspeaker is only going to be $\approx 10\%$ efficient
But bare in mind physicists like to break everything down into simple models and real world acoustics is anything but simple. Look into turbulance to find out more about that sort of thing, if you like.