This is a good (and notoriously difficult) question. I'm going to follow the explanation given by Crispino, Higuchi, and Matsas in their review 0710.5373, but you should be aware there are different answers out there and also there is no (uncontroversial) experimental test of this effect.
Having said all of that, the basic picture I have (and is given in the review, especially section III.A) is that the inertial observer does not see any thermal radiation. The accelerated observer sees thermal radiation, and so if the accelerated observer has a particle detector with them (in the review this is taken as a two state system, with a ground state and an excited state), the thermal radiation can cause the particle detector to become excited. From the perspective of the inertial observer, these transitions do not occur because of any thermal radiation--the inertial observer doesn't see thermal radiation. Instead, the acceleration causes the detector to interact with the vacuum in a way that can cause a transition in the detector. More or less, the inertial observer sees the particle detector experiencing a time-dependent hamiltonian because of the acceleration. The inertial observer sees the accelerated observer putting work into maintaining their acceleration, and this work is the ultimate source of the energy causing transitions in the detector.
To state things in more direct terms, the Unruh effect is often taken to mean that if we accelerate a pot of water, it can boil. In the accelerated reference frame, this happens because the water experiences the thermal bath. In the inertial frame, this happens because the accelerated pot can interact with the vacuum, with the energy ultimately coming from the work needed to accelerate the pot.