Starting a physics Ph.D. at 60? Decades ago I was in a physics Ph.D. program, made a hash of things, and quite rightly was dropped from the program. Since then I've had a very satisfying career as a computer programmer, and picked up an M.S. in Applied Math. I'm currently working in a bioinformatics research lab, and really enjoying myself.
I think I might want to retire in five or six years, and lately I've been daydreaming about returning to my first love, and entering a physics Ph.D. program. This would be entirely for my personal satisfaction. I have no intention of seeking a further career in physics, I'd just like to spend a few years working in a physics research group. I've been out of physics too long to really know what area I'd like to work in, but I'm thinking quantum computing or quantum optics.
While I wouldn't require any financial support, I recognize that taking on a Ph.D. student is still a major investment of resources for a researcher. My hope is that my years of experience in software development would be enough of an asset to make it worth their while.
I'm just curious if there is anybody in the audience who's done something similar, or any faculty who can give an opinion about whether this is a realistic notion.
Update:
Thanks to all for your answers and comments. I think nibot's comment that the thing to do is to establish relationships with some faculty I might like to work with is the most "actionable" answer. I really appreciated all the other anecdotes and advice though.
 A: I would advise to test the waters first. Are you really motivated enough to spend the long hours, days, months? Do you have the stamina to try the next path given that the previous zillion you tried led to nothing? 
ArXiv provides a wealth of recent papers on any physics subject (including quantum computing and quantum optics). Make sure you keep up-to-date in a few areas of physics that appeal to you. If you manage to do that in your free hours, chances are you will enjoy your PhD work. It will also help you making an informed choice on the subject of your PhD. 
Good luck!
A: The point of PhD programs is to produce future scientists; it seems unlikely that any decent department would admit someone who indicates that they "have no intention of seeking a further career in physics."  
Since you do have a background in physics, math, and computing, it does seem like you could earn your keep in a physics lab.  Assuming you are interested in experimental physics, I suspect your best bet would be to get a half-time job in a physics lab with an informal agreement that you can also audit the department's courses.   Actually taking a spot in a formal PhD program would probably be seen as unfair to the person you would be displacing, who would actually need the PhD for their career.
Theoretical physics would be more difficult, since it is much more difficult to "earn your keep" as a novice theorist.  I think it would be difficult to find an advisor in theoretical physics who would have the patience to mentor someone who explicitly did not intend to continue in the field--unless that person was manifestly brilliant.
A: Your CV is very similar to mine, although I made my career in supercomputing. I did have one job which was close to physics (doing programming for a plasma physics group). Perhaps if you volunteer yourself to a research group (as free or cheap labour) you can become part of the research effort without having to jump through all the academic hoops. At my age, I'd rather be involved in teaching the next generation, as learning neat stuff, but not having a use for it myself seems a waste.
A: The second tier schools offer "terminal masters" physics degrees. Some of these are intended for night students, for example, high school teachers that want an MS to get a better pay rate. For example, U. Washington offers such a program, along with a graduate non-matriculated student admission:
http://www.phys.washington.edu/grad.htm#gnm
If you show up as a non matriculated student and then crush the classes, I suspect that they'll let you in as a grad student. If not, you'll have recent grades showing your ability when applying to a different school.
A cool thing about not being in physics grad school is that you can study on your own. To do this requires a lot of drive. I suggest looking deeper into those areas of physics which you find absolutely fascinating. Don't make it "work", make it "play". Do this a while and soon enough you'll find out that you're a world expert in something. Write papers on it. Publish them. I think this is the right way for older people to edge their way into physics grad school. The idea is that you give yourself a reason for the sacrifices needed.
Unless you're completely driven, it's difficult to teach yourself physics on your own. So don't worry about giving yourself a complete graduate physics education. Concentrate on the stuff you're particularly fascinated by. If you decide to go to grad school you can always learn the other stuff in school.
To get into US grad school typically requires that you take the GRE. Go ahead and pick up one of the cheap books on the subject and see how you do. The general exam has three parts, quantitative, verbal, and writing. You should get a 750 to 800 on the quantitative as it is just high school algebra. Less than that and you should study / practice the subject. Make it into a video game. Older people do fine on the writing and verbal sections.
Then there's a physics GRE. There are sample tests for that, too. I'm 53 now. I also decided it might be nice to complete a PhD in physics. So I took the Physics GRE last October and maxed it out-990. This is way better than you need to get into physics grad school. So far, I've been rejected at the schools I applied to, but I only applied to the best.
As far as getting a physics PhD as an old man, yes I have no doubt it can be done. But you may want to apply to schools that will be impressed with your credentials.
A: Whatever you do, do it for fun.  
I've a similar age, graduate in electronic eng. and I've make a living on IT. I have a passion for physics and cosmology, born 20 years ago, when a friend (also electronic eng.) make me see interesting things untold in the textbooks. IMO new brilliant viewpoints. Now I have a physical explanation to all darkness: DM, DE, BH, even genuine Ufos, as described by pilots. You may smile, doubt and vote down but give me the benefit of the doubt until further explanation. As an example: If a new particle,  Z0 like, makes it appearence near 780 Gev range (LHC - 2011 ?)  I will know that the prevision of Douglas Pinnow (no one is reading his book) was right (the next one circa 1450 Gev). For now SUSY seems in trouble ( "Beautiful theory collides with smashing particle data" ) Lets wait and see. 
Another 60yo electronic engineer, and friend, is pursuing an Astronomy graduation. I think that I've more pleasure in studying by myself, asking and searching, without classroom, exams, schedules.  
Bioinformatics is awesome and a lot of challenging work is yet to be done. I wish I had the time for study it.
A: After getting a Engineers degree from Naval postgrad in his youth and working twenty-five years as a industry researcher, my father went back to school and got his Piled Higher and Deeper in M.E. a few years after I finished mine in physics{*}, and I had a grad school colleagues in his late fifties.
Neither seems to meet any unusual obstacles or special resistance.
I would, however, recommend talking to the department head and some prospective advisers before putting in any paper, just so you don't catch them by surprise.

This generated two funny episodes


*

*He called me to ask for help with his homework

*He was the session chair at a conference to which his adviser was submitting a paper.


and, of course, I had to bring my regalia to his graduation for the Doctor and Doctor McKee pictures.
A: I quit computing in the late 1980s as a result of an early mid-life crisis in my mid 30s. After a few years getting myself back together, I ended up in an M.Sc. at the University of Durham, UK, paying my own fees and for my food and lodging. Up to this level is easily manageable, because it's essentially just learning. What can make even this level not work out is what makes the next level more difficult to negotiate — real research.
With some life experience behind me, and having read around quantum theory and having come to naive opinions about what was "wrong" with quantum theory, I almost didn't finish my M.Sc. project, and I was only just adequately prepared for the examination component, because I went off on an almost unguided goose chase after a particular piece of mathematics (Clifford algebras, a pied piper if ever there was one). One academic managed to keep me from going completely off the reservation. I've chased after many other pieces of mathematics in the 20 years of research since, so I have a certain perspective on realizing that I've wasted another year or several on another piece of mathematics.
What an older person can bring to research can be a real asset and a real pain for an academic to have to deal with: bull-headedness. If you're older than the academic who is guiding your research, you nonetheless have to take their guidance, at least up to a point. When an academic looks at you, they will be choosing you over someone younger who is probably also more malleable. An academic has to decide whether they want someone who will be no trouble (and probably also not produce great research of their own in the future, but nonetheless be smart enough to help the academic's research go forward nicely, which is a delicate equation), or be a lot of trouble (but the academic can hope for and look forward to a satisfying experience of seeing their mentoring bear fruit many times over). Of course an academic's ego is a fundamental part of who they choose to take as graduate students. It has always seemed to me that this calculation is generally rather different when an academic considers a mature student. Being a real mentor to a graduate student is exhilarating and exhausting if they're really good at research, but could be worse than pointless if they react badly when they discover they're not. 
Twenty years ago, I wanted to make some academic let me do things my own way. That didn't work out (!), because I couldn't find an academic who would put me in the place they had instead of taking someone much more promising-looking and just graduated. Putting a brave face on this, I've sometimes thought that it would have been immoral to take those places from the young people who beat me out of those places. Consequently, I've pursued research independently, with an M.Sc. but without a Ph.D., which can be done if you're persistent and you can find a way to get good access to an academic library. Although you will make much better progress in research if you can find a good real-life community, I think it may be better to spend your money on access to a library and to find effective ways to be part of the on-line community (giving as well as taking, of course).
[My qualification for these observations is somewhat sideways: I'm married to a Classics professor, so I see from quite close up, but still from secondhand, and in an arts subject, the process of becoming an academic researcher, which many promising undergraduates never manage. Second-guessing who will make the jump is a large part of taking someone on as a graduate student. Needless to say, these comments are bashed out in a half-hour, they're incomplete, they're not gold dust, and you're not me. Good luck.]
A: In the UK there is the Open University which is a distance learning organisation with interactive aspects. As well as actual degrees it is possible to sign up for (sets of) modules as part of a longer term MSc (Master of Science) say. So if something similar is available in the US you might want to prepare by choosing some advanced MSc type topics to study in this way. The OU takes students of all ages, as I am sure any equivalent system would.
