From this link below " Refraction is the bending of waves because of varying water depths underneath. The part of a wave in shallow water moves slower than the part of a wave in deeper water. So when the depth under a wave crest varies along the crest, the wave bends.
So my guess is its due either to variations in water depth or else it's an optical illusion. If you can check it easily, I would bring a measuring rod, and do measurements all around the area below where the wave bends.
Conservation of energy is paramount here (as you know), so no matter what we think we see, if you hold your boat at constant velocity, in reality, that wake is going to dissipate sideways only, never backways toward the boat, over time. So unless there is another force present to give the wave more "kick", the bending must be an optical illusion, no matter how many observers agree that they saw it.
So what possible forces could push the wake back towards the boat? Normally it would spread out wider and wider as the boat moves further away, but to change direction back towards the boat needs a force and unless the change in depth idea is true, see the link above, there is no other force to cause this reversal in direction. If the depth of the lake is constant, that it adds support to the illusion idea.
I have no experience of water wakes, (other being a passenger on ferries and rowboats) but it does seem to me that water waves, like mirages and rainbows, are a much likely cause of optical illusions that anything static, merely because there are just many more variables involved.
So to sum up, unless you can test the depth to ensure it's constant, and make sure that no other forces, such as a wind blowing towards the boat, are involved, I would make put my money on the illusion explanation.