Although it may not look like it at first glance, this problem is actually valid. In particular, its statement that "the cross-sectional area of a parallel beam of light is Lorentz invariant" is actually correct, in that a parallel beam of light does indeed have a Lorentz-invariant cross-section. And in any given inertial frame of reference, the cross-section involved is indeed just an ordinary purely spacelike cross-section. And in any given inertial frame of reference, the cross-section involved is orthogonal, in an ordinary space-like sense, to the direction in which the light is travelling.
The above paragraph requires some further elaboration, or the paragraph will run the risk of it, too, appearing to be wrong at first glance. But first, I'll clarify a couple notational issues in the problem statement that have the potential for causing confusion:
In the convention Padmanabhan is using, roman indices include all four dimensions $0$ through $3$, instead of roman indices referring only to dimensions $1$ through $3$ as they do in some conventions. I.e., the hints purely deal with four-vectors, and always deal with all four components of the four-vectors. Also, although a Lorentz transformation plays a central role in this problem, the hints always only use one coordinate system. Or as an alternative to the above sentences, it works to view the problem statement as using abstract index notion.
The use of primes on $a'^i$ and $b'^i$ may appear to contradict the above statement about the hints never using primed components, but it doesn't. The primes on $a'^i$ and $b'^i$ do indicate the used of a "primed" coordinate frame, but not in the way one might think. $a'^i$ does not denote the primed components of the four-vector whose unprimed components are $a^i$. $a^i$ and $a'^i$ are two geometrically different four-vectors. $a^i$ is basically a displacement vector that runs alongside the edge of a square which is orthogonal to $k^i$, and is purely spatial in the unprimed frame. $a'^i$ is the corresponding displacement vector on the projection, along lines parallel to $k^i$, of that original square onto a surface which is also orthogonal to $k^i$, but is instead purely spatial in the primed frame. Similarly with $b^i$ and $b'^i$.
What's likely the biggest potential source of confusion in this problem, however, isn't a notational issue, but rather a conceptual issue about the nature of the cross section. I was careful in my first paragraph to say that the beam of light "does ... have a Lorentz-invariant cross-section", instead of saying that the cross-section is Lorentz-invariant, because there are two different "natural" ways in which one might define an orthogonal cross-section of a parallel beam of light. In an inertial frame of reference in which the beam is stationary, the two definitions are equivalent, but in a frame of reference in which the beam is moving, the difference between the two definitions is crucial.
Consider first the beam in an inertial frame in which the beam is stationary, and think of how you would go about defining an orthogonal cross section of the beam, just in a three-dimensional sense. One way would be to consider the set of all locations where the beam exists at some given instant, and then consider a tangent vector from the light's source to its destination along the surface of that set of locations. An orthogonal cross-section can be defined as the intersection of the beam with a plane that goes through some given point, and is orthogonal to that tangent vector.
But another way to define the orthogonal cross-section would be to consider the velocity vector of a photon in the beam, and then to define the orthogonal cross section as the intersection of the beam with a plane that goes through some given point, and is orthogonal to the velocity vector.
The tangent three-vector and the velocity three-vector used above are parallel, and can be considered to be the same three-vector if they are appropriately scaled. However, those two three-vectors are really the spatial components of a couple of four-vectors, and those two four-vectors transform differently under a boost. The four-vector that corresponds to the tangent three-vector is a purely spatial four-vector, i.e., it's a four-vector whose time component in the inertial frame in question is zero. But the four-vector that corresponds to a photon's velocity three-vector is a null vector, with a non-zero time component. If you transform the two four-vectors by a Lorentz boost to another inertial frame, and then take the spatial components of those transformed four-vectors, you will get two different three-vectors in the other inertial frame. All the above is really pointing out is that if a beam of light is moving, the velocity of the photons in that beam will not be parallel to the beam's surface at a given instant.
In the above, the cross-section defined by considering the beam's tangent vector does indeed undergo a length contraction under a boost. It's only the cross-section defined by considering the light's velocity vector that has an invariant area.
As to the remaining point of confusion, using $u^i$ and computing $\alpha$ and $\beta$ isn't really a red herring. All expressions use components expressed in the unprimed frame. To ensure that $a'^ia'_i$=$a^ia_i$ will continue to hold in the primed frame, you need to ensure that $\alpha$ and $\beta$ are Lorentz scalars. As you've discovered, it turns out that you don't wind up really needing the actual values of $\alpha$ and $\beta$, all you care about is just that they are Lorentz scalars. And if you determine $\alpha$ and $\beta$ as per the hints, you'll see that they are indeed Lorentz scalars.