Is it possible to restart formal higher education in physics at a later age? I would begin by apologizing for a soft question. Reading the FAQ, I have tried to formulate my  question precisely in the last paragraph and you may skip the following paragraph where I make subjective observations, which I think may be relevant and any comments would be more than helpful. By formal education I mean registering in an academic department in a university as a grad/Phd Student.
Subjective belief: I feel I am personally not upto the mark of high-quality physics students. I need to learn a lot more, and do a lot more problems to be any good. I've noticed that Physics particularly is a very selective field dominated by hard-working and primarily young people who accomplish a lot at an early age (olympiads/contests, early phd's from very good places, etc). Even among this bunch a lot get filtered out and wouldn't get faculty positions. Theoretical Physicists tend to be outliers in a field almost entirely composed of outliers. And to learn from such a community, I feel an academic environment is essential, and not just reading books in isolation. So I beleive academia is the only legitimate environment that functions to produce sane physicists who know their field. However, if one has been set-back for some reason but carries the self-beleif that he can be smart enough to do it, would ageing be a parameter. I read a statement in Kleppner's undergrad mechancis book that "by finishing all this at the age of 26, Newton set the trend which follows today, that great advances are made by young physicists". I haven't found a counterexample so I would trust academic departments have the same criterion.
Q: Do people who missed out all the action in their 13-20 ages miss out on having a good career? What are the possible reasons to take a hiatus of "self-study" for a 22-year old so that he may be a good candiate for a respectable physics career (Phd/postdoc etc) later. 
 A: Remember the famous people are outliers. Superstars - just like in Hollywood - are not the norm.
A lot of science is done by very bright, hard working but ordinary people, the same very bright ordinary people that work in industry.
Yes there are fields, particulalrly maths, where there may be a biological reason that people do all their best work before their brains hit 25 - but it's not true in a lot of fields. It takes a lot of years to get the experience to know how to do things in many branches, especially experimental ones, and even more years to get up the tree to a position where you can get do to the experiment.
BUT taking 10years out because at 18 you don't think you know as much as everyone else is equally silly, most of your peers don't know as much as they pretend either!
A: It depends on the country. In Greece there are so called "open university" academic programs,  programs geared for older people wanting a higher education. Last week a 91 year old man was interviewed on TV because he is pursuing studies for a degree in "art appreciation".
I would agree with Martin Becket , that if one is serious enough in wanting to study physics one should not put it off for "not good enough in background" reasons.  Yes, it is a competitive field, still, like any race, it is best to run it and see the results than not to and regret it. 
A: Taking a short hiatus should not prove fatal to your futur career, whatever that may be. I believe the theorist Dijkgraaff went to art school halfway through his studies! You're still very young, and the idea that all good work in physics and math is done before someone's 25th birthday seems farfetched; any major experiment takes years to set up, and good idea's in theory also take some time (the name of Mischa Gromov springs to mind). However, if you take a break, do it for the right reasons. It's not very likely that after one year, you'll return as the head of your class, full of wisdom about Chern classes and fermion doubling.
Finally, 'ageing' also differs per country. For example: in France, people tend to finish PhDs really early (around their 25th), but in Germany it's quite common to still be working on your Master's at that age.
A: I went to your profile and looked at some of your questions and answers. You should stick with it, I hope you are. You are bright and insightful. Don't get discouraged. You are smarter than you think. If you are suffering from chronic low self esteem, you might be suffering from depression. Consult a professional, and don't have too much pride to get help, even if it means drugs. A person as smart as you should not be halting an academic career out of self doubt regarding ability.
