Why was science before relativity postulating three space dimensions and one time?
Because that was the distillation of observations over millennia. Once geometry was established as a mathematical discipline/model it was the simplest way to describe/fit observations.
What are observations/measurements?
They are the way our five known senses interact with the environment and bring information to our brain, which organizes it for retrieval. Writing and arithmetic extended the organizational pattern so that information could build up over generations.
The technological advances after the 18th century increased the possibility of observations and the advances of mathematics increased the complexity of fitting the observations/data with the aim of having as simple a mathematical model as possible.
Dimensions kept being 3 space ones and one time, the space ones a vector space with a length well defined for vectors.
The data began to be complex enough that in the beginning of the 20th century special relativity was needed to describe the data, and it needed time to be part of a pseudovector space with a length defined in this space. Special relativity fits the observational data extremely well, in nuclear physics and particle physics.
In parallel the forces observed acting on particles were organized in a coherent manner in a coherent mathematical model . These forces at the particle level are interactions of point particles with each other, and complicated mathematics describes accurately the observations. Then the search of physicists for unifying theories, i.e. mathematical models that unified all the interactions/forces, led to higher dimensions.
As you state we do not see with our five senses any higher dimensions. Theorists are smart though, so in the theories they postulate ( like axioms in mathematics) that the other dimensions cannot interact with the particles we have observed and classified, except through the gravitational interaction. Their smartness extends to postulating that those extra space dimensions are curled up, like little circles going around, in so small dimensions that we would never see them even interacting gravitationally with them. This, because we have not seen particles disappearing and leaving four dimensional conservation laws hanging/violated.
meaning you could stick your hand out and be in two dimensions at once?
Curling up the extra dimensions means that you and I and everything , even though we exist in those, our hand will never interact and tells us that they exist, as it tells us that a third dimension exists outside a window.
That also doesn't make any sense. So, how can string theory claim there are more than 3 spatial dimensions in our world?
The claim is a well formulated theoretical hypothesis which is being tested with higher and higher energy experiments at LHC at CERN . There are predictions of superstring models for the experiments at LHC, and if the predictions pan out then one can take the higher dimensional theories seriously and push for more and more experiments to see what is up.