I want to learn electrical engineering on my own, specifically because I'm interested in loudspeaker design, more specifically how to design active dipole loudspeakers using DSP crossovers.
I have found resources to teach myself the necessary mathematics, however I don't have a solid sense of what undergrad level physics I will need, much less how to learn it on my own.
Any pointers or suggestions are greatly appreciated.
Before I start, I should say, I am yet to meet a physicist who has a clue about the practical aspects of loudspeaker design. Likewise about audio engineers ;-) so the fact you are asking this, means you are well set up for making some good speakers.
John Bird has written an electronics textbook which could be very useful for you which covers all the introductory electronic engineering theory to get an idea of what its about. Great way to get started, but maybe too simple for you. Have a look.
Undergrad Electronics is actually not so deep as it is broad. So, if all you want to understand speaker design you actually won't need a lot (to get started anyway) and actually making something that works, which is always a good place to start.
There are many very good textbooks for electronics and loudspeaker design knocking about and the core principles from decades ago still apply today, so that is lucky. There is also a electronics stackexchange site if you have specific questions. Don't be afraid to look at the older textbooks. Quite often they have a lot more core principles in them then the modern books and the added bonus that they are much cheaper on the whole.
There are a plethora of related coursera.org courses on the topic so that might help you along.
You will need to know some tricks like nodal analysis, applying Kirchoffs laws, Thevinins theorem, applying Ohms law (obviously), small signal analysis (for operational amplifiers) and if you are going digital, logic gates, boolean algebra all the books on digital electronics tend to be the same more or less. Perhaps you already know that much already without me telling you so I guess where it gets more strictly into the physics is when you consider things like the quantum hall effect and electromagnetism in general (try Griffin), you will need to understand fourier series and fourier transfer and waves in general -how they behave at boundaries, how sound travels through the air, resonance the wave equation. It wouldn't hurt to learn a bit of thermodynamics. Your carefully crafted loudspeaker is only going to be $\approx 10\%$ efficient
But bare in mind physicists like to break everything down into simple models and real world acoustics is anything but simple. Look into turbulance to find out more about that sort of thing, if you like.
As someone who studied electrical engineering in undergrad, I don' think you need much physics as such. If all you're interested in is designing speakers, there are basically 2 different approaches.
What you should be doing, in theory
You should learn quite a bit of math, at least to understand fourier/laplace transforms and their discrete time brethren. Knowing some linear algebra is crucial and basic linear system theory is very important. Once you have an understanding of that, you should be able to learn some DSP. I would like to recommend the book on Discrete-time signal processing by Oppenheim, Schafer and Buck. I think it gives a good idea of DSP, along with a very intuitive understanding.
To build active circuits, you actually need to learn how to build analog circuits. I don't know of good textbooks in this area as I'm not really interested in this area. I can however suggest some very well regarded lectures and course material available online. You might find resources from the Analog Circuits course and the associated lab course useful.
The way to actually get something done, in practice
Go get some electronic components and build simple speakers. Search the internet/books for better and more involved designs and read up on what the improvements are and try to keep improving your designs. One of the best methods is to actually se your speakers and identify a specific problem in them you'd like to solve. Then search for a way to fix them and try to understand why that works. Keep iterating and that should take you a long way.
The former approach would teach you all that you need to know before you actually build anything of use. In the latter approach, you learn by actually making stuff. Since you're interested specifically in speaker design, I would recommend the latter approach, augmented by a little bit of the former as and when the need to understand some component arises. Often, the people who can get things done in practice can do with a passable and rough idea of the background theory, and the people who concentrate too much on the theory can rarely get things done in practice.