What you propose is a version of concentrated PV.
Concentrated PV has been in development for some time.
The big issue with PV, is lifetime cost per unit energy. There are no other constraints really - there are plenty of alternative commercial panels that material shortages aren't an issue; we know we can manage their exogenously variable power generation; and there's no shortage of land to site them on. So it's all about cost.
Adding optic fibre to a system is going to lift the costs.
So, the concentrated PV systems that have been looked at, tend to work on the basis of very cheap lenses, and then very expensive PV. Because, given the intense power per unit area of the sun's insolation ($1kW/m^2$ at sea level, peak sun), when you start concentrating it to ten or a hundred suns, you need some pretty remarkable materials to be able to cope. You might be looking at efficiencies of 30% or so, so with a hundred suns concentration, then $1m^2$ of CPV is going to have to dissipate $70\% \times 100 \times 1kW/m^2 = 70kW$ of heat
Optic fibres would break that cost model really: you'd have an expensive lens system, and then an expensive PV cell.
So, the answer is about economics really, rather than physics, but it does tie in to the extreme material physics going on in concentrated PV cells.