Let me first say that I'm a fan of this theory, so whilst I'm giving what I believe to be a neutral response, bare in-mind that I'm pro-Bohmian which is in many fields an atypical viewpoint. To respond to the core question, 'why would a pilot-wave theory be wrong?':
There are plenty of reasons why 'a' pilot-wave theory could be wrong, but in terms of serious interpretations of QM that ask 'why not particle AND wave?', Bohmian mechanics (BM), otherwise known as de Broglie-Bohm theory, quantum hydrodynamics (sometimes), or pilot wave theory, is really the only game in town, so I'll respond to this specifically; please say if you'd like a more general response, but I assume this is what you're interested in.
First a little historical context - I won't go into too much detail because this is really more a topic for https://hsm.stackexchange.com/ but it helps put things into perspective so I don't think is off topic.
Louis de Broglie was the first to come up with the idea of pilot-waves (some might argue it was Erwin Madelung - he did devise almost the same mathematics earlier, however never to my knowledge considered them conceptually as de Broglie did) to solve the issue of wave-particle duality, which had been the subject of his 1924 thesis (and would win him a Nobel prize in 1929). He presented his idea at the 1927 Solvay conference, alongside presentations of the Copenhagen interpretation (what we now think of as standard/textbook quantum mechanics), and Einstein's view that QM wasn't a complete theory. It wasn't well received; especially by Niels Bohr (who had already won a Nobel prize while de Broglie was still in grad school), who is perhaps one of the biggest reasons Copenhagen is the standard interpretation. He was a very vocal proponent of Copenhagen and had a history of shooting down people who disagreed with his views. Hugh Everett (the creator of the many-worlds interpretation) had a similar interaction with him much later which ended in Bohr's description of Everett as being 'undescribably stupid and could not understand the simplest things in quantum mechanics'. This perhaps gives an idea of his general attitude. His main objection to pilot-wave theory however seems to be (aside from being an affront to his own idea) that he believed QM was new physics, and anyone attempting to explain it in real terms (because particles always have a definite, real, position in pilot-wave theory) was kidding themselves and unable to let go of various preconceptions.
So, after 1927 de Broglie essentially abandoned his theory and Copenhagen became the standard view of quantum mechanics. John von Neumann didn't help the situation when in 1932 he came out with a paper that would rule out de Broglie's pilot-wave theory as false. Grete Hermann quickly proved von Neumann incorrect, however her work remained in relative obscurity until the 70s, so until 1966 (when John Bell disproved von Neumann in the same way) many physicists (who cared about the problem) falsely believed a pilot-wave theory was not possible.
In 1952 David Bohm; unaware of de Broglie's earlier work, reformulated pilot-wave theory. Unfortunately by this time he was largely being shunned by the scientific community due to his communist affiliations, so again his work didn't prove popular; it is only very recently (looking at published paper metrics, around the year 2000) that this theory has started to gain traction, and especially since 2006 when physical pilot-waves, which have already been discussed here, were discovered that were able to demonstrate some behaviours previously thought to solely be the domain of QM.
So, that's some reason why the theory has often been ignored until now, and why many people and textbooks dismiss it; often it is falsely considered to have been proven wrong, or it just isn't on people's radars (Viz. if you have a working model of QM, why do you need another one?)
These days the criticisms BM receives are generally philosophical objections such as its surrealist trajectories or s-state electrons being time-invariant. These are mathematically sound though and have not been disproven; my personal view is complaining that quantum mechanics doesn't work like you expect it to, or doesn't act like a classical system, has already been flogged to death - we know quantum mechanics isn't 'normal', so why should we suddenly expect various trajectories to work in the same way as say, pitching a baseball. This is a common objection nonetheless, and interestingly the exact opposite to Bohr's original objection; that pilot-waves were too classical.
Next are Occam's razor arguments; that BM adds complexity without giving anything in return. There is possibly some truth to this, however the guidance equation is derived from the Schrodinger equation and conceptually isn't too far removed. Certainly Copenhagen has its own problems when it comes to this, and one could argue that a theory such as superdeterminism is simpler than them both. On the other hand, the extra maths of BM does have some use in a few fields like quantum chemistry, where it provides more efficient ways of solving certain problems than through the maths of standard quantum mechanics (SQM).
Finally there are arguments about BM not being compatible with relativity and QFT. It's important to be aware that BM is a non-relativistic theory. There are extensions to it though that do in fact incorporate relativity/QFT, and you can find this discussed in various literature e.g. https://arxiv.org/abs/1205.1992. It certainly isn't as mature as SQM is, so this can be an argument again for SQM, however it certainly shows that this is not a failing of BM, and the maturity of SQM is really its advantage here (BM has had only a handful of people working on it seriously for about a decade).
At the end of the day (and this is perhaps the important takeaway), all experimentally verifiable results are identical in BM to all other not-disproved interpretations of QM; they are all as right as each other, so all objections are philosophical in nature or have historical context. That doesn't mean the situation will remain this way; certainly it's possible that experiments will be devised that prove or disprove various current interpretations, and of the various interpretations my money would be on BM being one of the most likely to, if it is wrong, be ruled out by experiment in the future.