I think the confusion here stems from different types of acceleration: coordinate acceleration and proper acceleration. Coordinate acceleration is essentially the second derivative of position relative to the frame of an observer (so objects that you think are following curved paths have a coordinate acceleration), while proper acceleration is the acceleration that would be measured by an accelerometer.
For example if you are standing on the ground, your coordinate acceleration is zero relative to the Earth's surface. But an accelerometer would measure the acceleration due to gravity, so your proper acceleration is $9.81$ ms$^{-2}$ downwards.
Say there is are two spaceships floating in empty space. Both their coordinate accelerations and proper accelerations are zero. Now say one spaceship (A) switches on its rocket engines. Someone in the other spaceship (B) would measure a non-zero coordinate acceleration, since the velocity of A is increasing relative to B. Inside A, objects would also be pushed downwards as soon as the rocket engines are swithced on, and an accelerometer would measure an equal proper acceleration $a$.
According to Einstein's equivalence principle, being inside the spaceship A while it is accelerating at $9.81$ ms$^{-2}$ is indistinguishable from being inside the spaceship while it is sitting still on the surface of the Earth. The proper acceleration of both 'frames' is the same.
Let's say that, from my point of view, another observer is accelerating. Now, from his point of view, he is standing still
If I understand your meaning correctly, he is (1) standing still from his point of view if he is not experiencing any proper acceleration. In the language of general relativity, this means he will follow a (timelike) geodesic through spacetime. If the spacetime around him is curved, the geodesic he follows could be curved, and his coordinate acceleration will be nonzero.
Alternatively, you might mean that (2) you are in spaceship B from above and he is in spaceship A (he is accelerating away from you). In this case, he feels a proper acceleration. If he realises that this is caused by him switching on his rocket engines, he will know he is accelerating. However, if he somehow convinces himself he is actually on the surface of the Earth, he might think that he is perfectly still. However if he was on the surface of the Earth, then he would still not be following a geodesic in spacetime because he is not in freefall - geodesics around the Earth fall inwards, following the path of a falling object with no air friction.
If the second paragraph (2) is what you were asking about, then I am still a little confused what you mean by "from his point of view, he is standing still" - I suppose everyone is always perfectly still relative to themselves, but that's a bit of a physically useless fact.
all he feels is an overall fictitious force of gravity
(1): He does not feel gravity, even though it bends his path (as measured by you), because gravity is not a force in general relativity. In fact by Einstein's equivalence principle, if he has no proper acceleration then he will no different from floating freely in empty space.
(2): He feels the acceleration that he is experiencing thanks to his engines (his engines are constantly changing his speed, which means he is deviating from a geodesic path). If he thinks he is actually on the surface of the Earth, he will attribute this acceleration to the fact that he is deviating from a geodesic path (which falls inwards towards the Earth's centre) by being still on the surface. Proper acceleration and deviation from a geodesic are the same thing (they both describe the same physical effect).
Now, if we accept this geometric view, It seems like acceleration cannot really exist in spacetime: every object keeps following a geodesic.
Although most inanimate objects in space have no proper acceleration, it is still possible to accelerate in space, for example by firing rocket engines or using a light sail. Also, the strong magnetic fields around some planets (as well as neutron stars) cause forces on charged particle in the vicinity.
So acceleration can definitely still exist.
When some external observer thinks that an object is accelerating, he should say instead that it is following a "curved geodesic"
Below I'll use 'accelerating' to mean something that has nonzero proper acceleration.
This is incorrect. A geodesic is the equivalent of a 'straight line' in curved space, and is defined as the path of a non-accelerating observer. Geodesics are 'straight' if the spacetime is flat, and can be 'curved' if the spacetime is curved, and the geodesic happens to not follow a 'straight line' in the coordinates that you choose to use.
If an observer is not accelerating, he will always follow a geodesic path (which could be curved) through spacetime. If he is accelerating, he will not follow a geodesic, but he may or may not follow a curved path.
Also see the wikipedia page on proper acceleration