Some preliminaries:
(1) Wormholes are a type of curved spacetime. Therefore there can't be any complete analysis of this problem in special relativity. You need general relativity.
(2) The basic logic is not that causality violation implies FTL, it's that FTL implies causality violation. There are other reasons besides FTL why you can get causality violation. One of them is if your spacetime has closed timelike curves (CTCs).
(3) Because this is GR, not SR, there is no unambiguous way to define the velocity of an object relative to some other, distant object. The question assumes that going through a wormhole does not involve FTL velocities, but in fact there is simply no way to define what velocity it would be.
There are straightforward arguments to the effect that if wormholes are possible, then CTCs are possible as well. The basic idea is that either mouth of the wormhole can be acted on by gravitational forces (I guess in the sense that it has a certain Komar or ADM mass), and therefore you can manipulate it and move it around (at least in theory, if you had the ability to manipulate huge amounts of matter). So you take one mouth, accelerate it away from the other, and then bring it back. This is very much like the twin paradox; the two mouths are no longer synchronized temporally. This kind of thing is discussed in [Friedmann 1990] and [Echeverria 1991]. See Echeverria's figure 1.
The Echeverria paper discusses some toy models involving billiard balls, and shows that, surprisingly, one does not always get causality violation despite the presence of CTCs. Echeverria was a student of Thorne at CalTech, and Thorne also gives a popular-level discussion of this idea in ch. 14 of his book Black Holes and Time Warps. IIRC Echeverria's paper proposed a research program to investigate this kind of thing in more detail, but as far as I know it was never followed up. This may have been because the research program didn't work out, or because Echeverria didn't get a permanent job doing relativity.
Friedman, Cauchy problem in spacetimes with closed timelike curves, http://authors.library.caltech.edu/3737/
Echeverria, Billiard balls in wormhole spacetimes with closed timelike curves: Classical theory, http://authors.library.caltech.edu/6469/