I'm not a physicist.
It seems to me that a quantum computer needs to store and access all its quantum states.
Currently the universe is concerned with calculating the interaction of its particles and radiation. If, via a quantum computer, we gave it a problem requiring enormous bandwidth/information-storage, couldn't this interfere with the universe's everyday housekeeping causing a general slowdown?
Of course it might be supposed that if everything slowed down proportionally then this would be undetectable. Is there anything we could use to detect a ubiquitous slowdown in the universe's metabolism? Could such a slowdown be catastrophic?