Is the gravitational force for an object on the surface of earth always pointed to the center? I thought it would be so because the mass inside the earth would attract us however they would be at a larger distance than masses on the surface of the earth.
For example, consider having a tall building near the object. Would the effective force vector of the object be pointing into the center of the earth or deviated?
 A: A large skyscraper's mass:
$$M=2.5\times 10^{8}\,\mathrm{kg}$$ 
For the benefit of the doubt, let's assume a centre-of-mass at the ground, and that you are standing at the building's wall, possibly 50 metres away.
$$r=50\,\mathrm{m}$$
The skyscrapers gravity on a $m=75\,\mathrm{kg}$ person is:
$$F_{g,\text{skyscraper}}=G\frac{mM}{r^2}=0.0005\,\mathrm N$$
Compare this with the $F_{g,\text{earth}}=mg=735\,\mathrm N$ gravity from Earth. The skyscraper is adding a perpendicular component, so the total force of gravity on the person is a vector, which now is slightly turned:
$$F_g=\begin{pmatrix}735\\0.0005\end{pmatrix}\,\mathrm N$$
The angle in degrees between this vector $\begin{pmatrix}735\\0.0005\end{pmatrix}\,\mathrm N$ and the original vector $\begin{pmatrix}735\\0\end{pmatrix}\,\mathrm N$ is:
$$\theta=3.9\times 10^{-5} ~^\circ$$
This is a very, very small deflection. Just the fact that gravity from Earth does not pull directly to the centre due to uneven density, centrifugal effects, altitude etc., might far outweigh this tiny deflection.
A: Geodesists typically distinguish between gravity and gravitation. Gravity is gravitation plus centrifugal acceleration and is measured by the acceleration of an object released at rest with respect to the rotating Earth, as measured from the surface of the rotating Earth.
A plumb bob points "down" -- the direction in which gravity (not just gravitation) pulls the plumb bob. The nominal angular difference between "down" and toward the center of the Earth reaches a maximum of about 0.1924° at about 45° latitude. This nominal angular difference results from the more-or-less ellipsoidal shape of the Earth. "Down" nominally points in the direction of the inward normal to the ellipsoid.
Even on eliminating centrifugal acceleration from consideration, leaving gravitation only, the resultant acceleration only points to the center of the Earth at the equator and the poles. The Earth's rotation results in a bulge of material around the equator, the equatorial bulge. The primary effect of this bulge on gravitation is called "J2". At a constant distance from the center of the Earth, the equatorial bulge makes gravitation stronger than spherical gravitation above the equator and weaker above the poles. The gravitational bulge also results in a component of acceleration that is orthogonal to the direction to the center of the Earth. This effect is greatest at about 45° latitude, where the angle between the nominal gravitation vector and the vector toward the center of the Earth is about 0.9°.
The Earth is not quite an ellipsoid. It has mountains and pockets of material the are more dense or less dense than nominal. (Mountains are for the most part huge pockets of material that are less dense than nominal.) These regions of varying density affect which way gravity (and gravitation) point locally. The angular difference between "down" as observed by a plumb bob and "down" as calculated by assuming an ellipsoidal Earth is called the deflection of the vertical. This deflection is very small, rarely over an arc minute.
A: The force vector would only point towards the centre of the Earth if it were spherically symmetric and non-rotating (except perhaps at the equator or poles). There are significant deviations from this and local gravity can deviate by tenths of a degree from a line towards the centre of the Earth.
e.g. see https://www.wolframalpha.com/widgets/view.jsp?id=e856809e0d522d3153e2e7e8ec263bf2
I believe that the rotational effects (which are easily predicted) are usually somewhat bigger than the effects caused by local geography.
These deviations were used historically to estimate the mean density of the Earth by Neville Maskelyne among others, who used the deviation of a plumbline caused by the mountain Schiehallion.
A: The assumption that the force of gravity points to the center of the earth is based upon the assumption that its center of mass is the center of a sphere, I.e., the earth is spherical and of uniform density. 
But since the earth is flatter (not a perfect sphere) there are some deviations in the direction of gravity because of that. Variations at the surface (mountain ranges vs plains near sea level) means it’s mass is not perfectly spherically symmetric and the distance to the center varies slightly. Those factors  can theoretically result in very slight deviations in the magnitude and direction of gravity.
Hope this helps.
A: The force vector would be deviated. That the gravitational force is pointed to the center of the earth is a simplification that assumes the earth is a point mass.
In fact all masses pull on each other, so the mass of your tall building will pull on the object, too, deviating it.
It might even be that the mass of the earth isn't equally distributed - quiet sure - and this causes some (very small) deviation.
