Choking at the attempts on creativity My question concerns the art of scientific discovery more than actual theories and laws of Nature. The question begs for discussions and subjective points of view since it's a personal one so please, just share whatever comes to your mind, no matter how biased your opinion is!
And the question is: How do you approach a task of devising an original theory/method/tool/experiment?
And specifically, what I'm intersted in the most is:

*

*How do you recognize a point where one should leave behind all the maps that have been already drawn and start creating your own?

*How do you avoid going in circles, constantly researching and never really getting to a point where you get creative?

*How do you decide if you should extend what's already been created or is it worth starting from scratch?

*If you go from scratch, do you go top-down or bottom-up, i.e., from sketch to working out the details, or systematically from the principles.

I know the question is very general, but I want to discuss it and see many points of view.
Why do I ask?
I'm a physicist. Not really active academically after I dropped out of PhD course, but I still like challenging myself with tasks that require sharp analytical skills, understanding of the matter I'm working on etc. be it car mechanics, programming or electronics. Being a physicist, I always loved to learn. It's been a great pleasure for me to discover the laws of physics or the principles of action of various manmade devices. But what I found out is that I'm struggling with getting myself to tackle problems that require letting go of what I know and going creative. I often get stuck on research which doesn't lead me anywhere, since I already know everything I'll ever need for a specific task. But when I get to the task, I get nervous, stressed and I look for various ways to draw my attention off of the task. It's kind of weird, but I'm glad I finally noticed it and I can start inspecting this peculiar worry of mine.
I'd like to learn about ways my fellow scientists handle with such matters. As I said beore, any opinion you have is worth sharing.
 A: In science the goal is not originality, but explaining the existing natural phenomena. Knowing the important scientific problems requires broad knowledge, whereas designing the ways to explore and explain these problems is where the creativity is required.
Remark: The question looks opinion-based, and I wanted intitiall to post above in the comments and recommend closure. But, seeing that the definite answer exists, I decided the question is not against the community rules, after all.
A: This is difficult to teach. It is the inductive part of science rather than the deductive, and as Hilbert points out it in Geometry & the Imagination, it is what characterises how science moves on.
The only method that I know of is to study what the best did beforehand. Unfortunately, this is often hidden by layer of formalism. For example, if we look at the definition of curvature it looks as far removed from the notion of curvature as it is possible to be. However, if we look at the definition of curvature that Newton & Liebniz used, then we see that they tackled the simplest case - a curve - and the definition was intuitive. In fact, they begin with an even simpler case. We take the simplest case, a circle. Obviously, its curvature is the same everywhere. Obviously, a circle is determined by one number, it's radius. Obviously, we should take it's reciprocal as a measure of it's curvature. Then not so obviously, we measure the curvature of a curve at a point by fitting a circle to the curve there, and taking it's curvature.
As we see from here, what we have is a sequence of quite simple moves. This is in fact what characterises the best scientific thinking, even if this does not look like it. Often the steps are elided, and often, the exposition is not so good, and often, the material has moved far from it's origin. Compare the above definition of curvature with
$R(X,Y)=[\nabla_X, \nabla_Y] - \nabla_{[X,Y]}$
The latter is unrecognisable from the first. Yet, it is rooted in the first. It is where it came from. Grothendieck used the analogy of the rising sea. Of a nut finally cracking open when moistioned by water over months. It also shows that building intuition is a slow and patient process that takes time to build and that it is an organic process. It's not unlike learning to play the guitar. First you fumble to fret the strings. The fingers ache. But after patient and regular practise over months and years, the fingers grow stronger, they know where to go, and you don't even have to think how to play - you play. Likewise with science, you have to build up the muscles of the mind.
I'd also add, that there are many institutional problems. The creeping corporatisation of universities so that they are driven away fron their mission of teaching, learning and understanding. The lack of contact that students have with researchers and teachers. This reflects wider societal orthodoxies in economic and social planning ...
A: 
How do you recognize a point where one should leave behind all the maps that have been already drawn and start creating your own?

When existing models don't explain some data.

How do you avoid going in circles, constantly researching and never really getting to a point where you get creative?

Simply abandon that line of inquiry once I recognize that indeed I am going in circles.

How do you decide if you should extend what's already been created or is it worth starting from scratch?

One always builds on the current framework of physics; since that is validated by experiment, any new idea must take the shape of this existing framework in the appropriate limit.

If you go from scratch, do you go top-down or bottom-up, i.e., from sketch to working out the details, or systematically from the principles.

I often get contacted by people that have left academia and claim to have found the one flaw in the theory of relativity, built a perpetuum mobile, or explained the universe in its entirety. I find it sad that people waste their time on such obviously non-productive re-inventions of the wheel when instead they could spend their productivity to try and solve an actual real-world problem, be that a technological challenge, or some more detailed description of some particular phenomenon.
A: Vadim gives the correct answer, but I think there is more to the question than that.
Science is the systematic search for knowledge. But setting up the system for searching is not science: there is a philosophical part (what is a system? what counts as valid knowledge?), there is a technical part (how do we acquire the data? how do we turn it into knowledge?), and there is a strategic part (what problems do we pursue? what problems do we drop? who do we join forces with to search?) In particular the question touches the strategy part: how do we prioritize research?
This is where the sub-questions of the question really matter. This is basically decision-making under high uncertainty: one can use theories like information foraging theory, expected utility maximization or other decision theories to try to optimize it, but in practice it comes down to personal judgement of what looks good. The exploration/exploitation dilemma is important and hard.
Some people have divergent thinking and generate a lot of ideas, but are less effective at converging on the best or implementing them. Others have few ideas but are good at implementation. Hence they should try to fit their approach to their mental style - or find a collaborator that complements their weak sides. Similarly different fields of physics may have very different "difficulty structures" where in some domains solutions to one problem often apply to many other problems, while in other domains each problem is very unique and there likely are few general principles. Depending on what the goal is, one may select the domain based on this - but often understanding the structure requires attempting some exploration or learning from domain experts.
