Why do some physicists believe that scalable quantum computing is possible? If you drop a glass cup on the ground, it will break and shatter into pieces. This happens all the time and is consistent with quantum mechanics. But it never happens that a shattered glass cup rearranges itself from the ground into someone's hand as a whole glass cup, even though this is also consistent with quantum mechanics. We see from this example that not everything that is consistent with quantum mechanics is possible.
As far as I know, scalable quantum computing has never been demonstrated either backwards in time or forwards in time. So a fortiori, I would think that this would be good enough evidence to suggest that scalable quantum computing is impossible. Yet, some physicists believe that scalable quantum computing is still possible. Why? 
 A: You might find this lecture by Scott Aaronson insightful.
https://www.scottaaronson.com/democritus/lec14.html
He basically makes a very convincing defence of quantum computing against arguments made by quantum computing skeptics. I certainly found it very interesting when I read it!
Also, your "entropic" argument is not entirely correct. A physical system will not change its state to one with much lower entropy spontaneously (except with overwhelmingly low probability). However, it can be brought to a state of lower entropy by performing work on it. This for example is what a refrigerator does!
A: I would say it is not "some" but "most" physicists who believe that quantum computing is in principle possible (although many think the engineering challenges will be very hard and have doubts about whether the resulting computers would be worth the cost).  The reasons are:

*

*The threshold theorem, which applies even to mildly correlated errors, e.g. see 1207.6131.
What about skepticism about whether the assumptions are met?
(a) I am not aware of serious proposals for error models that would violate this in the sort of systems being considered for quantum computing.  Note that there are error models that violate this in chaotic or turbulent systems, like plasma.  But these are separated by phase transitions from the systems we can build.
(b) We have very good small-scale systems, e.g. see 1402.4848.  Again no one has even a proposal for how this will fall apart when this is scaled up.
(c) If correlated noise stopped quantum computing, then it would also stop classical computers, which use very similar techniques of error correction (once you see each bit as being encoded in repetition code involving many electrons).


*Maybe quantum mechanics is false?  The problem is that it's been tested a ton over the last century in a wide range of regimes to a ridiculously high precision.  No one can propose a modification that even works on paper that would stop quantum computers but allow all previous observations.  One problem is that small modifications that add things like nonlinearity would lead to crazy consequences, like time travel.
In short, the hypothesis that quantum computing is possible is based on a foundation of (a) math, and (b) very well-tested observation.  I think it is comparable to the hypothesis that a human mission to Mars is possible.
A: The reason is Shor error correction. Shor demonstrated that by using 9 bits for every bit, you can reverse any decoherence event on any of the 9 bits by doing some measurements on auxiliary quantities. Before the existence of error correction, it was plausible to say (and Unruh did say) the quantum computers are unphysical, because they require no error in a macroscopic system. This is an impossible position to hold past 1996.
The error correction method has been made more efficient since, by Shor and collaborators, and the upshot is that if you make a small quantum computer which is coherent for long enough time, and you can encode some dozens of qubits robustly so that you can reverse the errors faster than they occur, you can scale up the computation indefinitely without problems.
This makes quantum computation feasable for sure, and there is no way to argue that it is impossible without arguing that quantum mechanics fails.
A: Demonstration in "Real Life". Others have given you many other sources, what more do you want?
Survey of NMR Quantum Computing
D-Wave, a commercial vendor
Quantum Information Processing with Trapped ions, (PDF)
A: Several prominent physicists are on record as being pessimistic about the prospects for unlimited scalability of Quantum Computing - See for example https://arxiv.org/abs/quant-ph/0311039
(I found that link when web-searching on "Gerard 't Hooft" and "sudden death", as I recall him speculating in an on-line lecture that at some calculation/complexity threshold, quantum computing devices would undergo an unaccountable but unavoidable "sudden death", the quantum equivalent of a core dump!)
A: Your information is slightly outdated. Quantum computing is possible. They even successfully implemented Shor's Algorithm to factorize a product of primes. In 2009.
