Let me first get something out of the way:
It seems to me that wood is an unsuitable material for this thought experiment in the following sense: wood is very resistant to change of dimension parallel to the grain, but not perpendicular to the grain. Wood changes very unevenly.
But of course your question can readily be reformulated to feature a material that also changes dimension dependent on how much water is absorbed, but evenly. Such even swelling is a better fit for comparison to thermal expansion.
When a material is heated, with corresponding thermal expansion, this expansion does not mean any internal tension will arise.
For absorption of water into a material I imagine the following: the material has structural integrity (otherwise it wouldn't be a solid), so I expect that when this absorbant material becomes saturated an internal tension will arise.
Presumably the material absorbs water because of forces of cohesion. I imagine that these forces of cohesion will cause water molecules to be wedged into openings that are barely large enough to acommodate a water molecule, hence development of internal tension. The point of saturation will be the point where the internal tension has risen to a point where there is an equilibrium of force. An equilibrium between forces of cohesion that tend to pull in more water, and force of internal tension that tends to push water out.
I expect that this internal tension will affect the size of a hole in the material.