I was stating at a hex screw sort of in awe of the physics underlying this rather simple mechanical system. It got me thinking, how would one go about determining the optimal shape for a screw to maximize its holding strength for a given volume of material? What maths and engineering concepts are required to solve it and has this already been considered?
Screws come in various types depending on their application. As @Nielsneilson said, they have indeed been optimized (approximately) over a long time, but the "fitness function" for optimization depends on multiple factors. You asked specifically about holding strength in a given material. The brittleness and elasticity of the material, as well as the coefficient of friction between the material and the material of the screw, are both important. If the concern is only that the screw may become "unscrewed" due to vibration or other such causes, the important factor is the friction. If the concern is that the screw may get pulled out of the material, breaking the material in the process, it is a more difficult problem.
An extreme case is that of screws designed for anchoring in drywall (plasterboard). Because drywall crumbles easily, closely space threads in a screw will simply turn the drywall to powder so the screw has essentially nothing to hold it in place. The solution is to shape drywall anchor screw threads a bit like an auger: a thin helical blade that slices a spiral path through the plaster, leaving intact most of the plaster that is within the cylindrical envelope of the screw.
Another extreme case is that of machine screws, designed to clamp two pieces of metal together solidly despite vibration. In that case, the material (the metal pieces or a nut) is plenty strong and the important thing is to be able to exert maximal compression between the pieces. For that purpose, it is advantageous to use narrow screw threads with a small helix angle https://www.afi.cc/blog/screw-thread-terminology. Larger thread depth for a given helix angle results in weaker threads, but stronger grip.
Given a specific fitness function (i.e., what does "better" mean in a given context?", it is fairly straightforward to set up an optimization procedure that selects the best screw shape for a specific application. For example, a genetic algorithm incorporating all the necessary factors and relationships can quickly find an optimal shape. (Transparency note: this link is to something I wrote about 25 years ago.)