I dont understand how one part of the equation seems more geometrical and compelling than the other, as they are both clearly components of tensors. (Perhaps I do not see what "geometrical" means?)
Exactly why Einstein felt that way (or even if it's historically accurate that he did), I'm not sure. But to give you an idea of why the two sides aren't quite the same, note that you can take any metric tensor that meets basic requirements and call it a solution of the Einstein equations as follows: Compute the various derivatives required to make the Reimann tensor and combine those to form the Ricci tensor and Ricci scalar. This gives you the left hand side of the equation. If that's 0, then you got a vacuum solution. If that's not zero, move the remainder to the right-hand side and call it a stress-energy tensor.
Now while that's procedurally correct, there's no reason to think that stress-energy tensor corresponds to anything physical. In general, it will not.
Going the other way is typically harder. If you start with a physically motivated stress-energy tensor, you may or may not be able to compute the corresponding metric in practice. The solution should exist, but finding it may be difficult. It's not the straight-forward procedural exercise that it was going in the other direction.
In either direction, the stress-energy tensor isn't so much a part of the geometry. Going "left-to-right" it is what's left over that cannot be explained by the geometric curvature. Going from "right-to-left" it's a source term that determines the curvature. In neither case is it a "geometric" object like the curvature terms on the left side.